Training & Optimization

Explore Training & Optimization through related topics and the articles other pages reference most.

Most linked

The Training & Optimization articles that other AI Wiki pages link to most.

  1. HyperparameterA hyperparameter is an input that configures a machine learning algorithm or modeling pipeline
  2. Learning RateThe learning rate is a hyperparameter that scales an update made by an iterative optimization algorithm.
  3. Loss FunctionA loss function assigns a numerical cost to a prediction, decision, or action after an outcome is observed.
  4. Supervised fine-tuningSupervised fine-tuning (SFT) is supervised training applied to a model that has already been trained.
  5. Gradient DescentGradient descent is an iterative first-order method for minimizing a differentiable scalar objective.
  6. Fine-tuningFine-tuning is the process of adapting a pretrained machine-learning model by continuing to optimize some or all of its parameters on data…
  7. Distributed trainingDistributed training is the practice of training a single machine learning model using many compute devices in parallel, splitting the…
  8. LoRA (Low-Rank Adaptation)Low-Rank Adaptation, usually abbreviated LoRA, is a parameter-efficient fine-tuning method for adapting a pre-trained model.

Explore articles

Browse subtopics (47)

Articles that also belong to these categories. Counts cover all of Training & Optimization.

Showing 1-60 of 169 articles

AdaGrad

AdaGrad (short for Adaptive Gradient Algorithm) is an optimizer for gradient descent-based machine learning that gives every parameter its own learning rate

Adafactor

Adafactor is an adaptive optimizer designed to reduce the memory used by second-moment estimates during neural-network training.

Deep Learning

Adam optimizer

The Adam optimizer (short for Adaptive Moment Estimation) is an algorithm for first-order gradient descent-based optimization of stochastic objective functions

AdamW

AdamW is a variant of the Adam optimizer that decouples weight decay from the gradient-based update rule, applying the decay directly to the weights instead of folding it into the loss as an L2 penalty.

Machine Learning

Alternating least squares

Alternating least squares (ALS) is an iterative method for fitting a product of two or more factor matrices to data by holding all but one factor fixed, solving the resulting linear least squares problem for…

AlgorithmsMachine Learning

Axolotl

Axolotl is a free and open source framework for fine-tuning and post-training large language models, written in Python and driven entirely by a single YAML configuration file.

Developer ToolsOpen Source AI

Bayesian Optimization

Bayesian optimization is a sequential, model-based strategy for finding the global optimum of expensive black-box functions in as few evaluations as possible.

Machine Learning

Clipping

Clipping is a family of techniques in machine learning that constrain numerical values to lie within a specified range or below a specified magnitude.

Deep LearningMachine Learning

Context Parallelism

Context Parallelism (CP) is a distributed training strategy that partitions the input sequence dimension of a transformer across multiple accelerators and uses ring-style point-to-point communication to…

AI Infrastructure

Convergence

Convergence in machine learning is the point at which an iterative optimization algorithm reaches a stable solution, meaning the loss function stops decreasing meaningfully and further parameter updates yield…

Machine LearningMathematics

Convex Function

A convex function is a real-valued function whose graph curves upward into a bowl or cup shape, so that the line segment (chord) connecting any two points on the graph lies on or above the graph itself.

Machine LearningMathematics

Convex Optimization

Convex optimization is the branch of mathematical optimization that minimizes a convex function over a convex set, a problem class with one defining advantage: every local minimum is also a global minimum.

Machine LearningMathematics

Cosine learning rate schedule

The cosine learning rate schedule, also called cosine annealing, is a learning rate decay strategy that lowers the optimizer step size from a peak value to a small minimum following a half period of a cosine…

Deep Learning

Cost

In machine learning, cost is the scalar number that summarizes how badly a model is performing on a chunk of data.

DPO

DPO (Direct Preference Optimization) is an alignment technique for large language models that directly optimizes a language model policy from human preference data, without training a separate reward model or…

AI Alignment

DeepSeek-R1-Distill

DeepSeek-R1-Distill is a family of six open-weight reasoning language models released by DeepSeek on January 20, 2025, alongside the flagship DeepSeek-R1 reasoning model.

AI ModelsChinese AI

DeepSpeed

DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.

AI InfrastructureDeep Learning

DiLoCo

DiLoCo (Distributed Low-Communication training) is a distributed optimization algorithm for neural networks introduced by Google DeepMind in November 2023 to train large language models across loosely…

Google DeepMind

Distributed training

Distributed training is the practice of training a single machine learning model using many compute devices in parallel, splitting the data, the model, or both across GPUs, TPUs, or other accelerators that…

MLOps

Domain adaptation

Domain adaptation is the subfield of transfer learning that adapts a model trained on a labelled source domain so it performs well on a related but different target domain, where labels are scarce or absent

Machine Learning

Dropout

Dropout is a regularization technique for neural networks that randomly sets a fraction of neuron activations to zero during training

Deep Learning

Dropout Regularization

Dropout regularization is a regularization technique for neural networks that prevents overfitting by randomly setting a fraction of neuron activations to zero on each training step, forcing the network to…

Deep LearningMachine Learning

Dust

Dust is a zeroth-order training method proposed by Q Labs in October 2026 for language transformers without backpropagation.

AI Research

Early Stopping

Early stopping is a regularization technique that halts the training of an iterative machine learning model as soon as its performance on a held-out validation set stops improving

Deep LearningMachine Learning

Elastic Net

Elastic Net is a regularization method for linear regression that combines the L1 penalty associated with Lasso regression and the squared L2 penalty associated with ridge regression.

Machine Learning

Empirical Risk Minimization

Empirical risk minimization (ERM) is the foundational principle of statistical learning theory: because the true risk (the expected loss over the unknown data distribution) cannot be computed

Machine Learning

FP8

FP8 is a family of 8-bit floating-point number formats used to accelerate deep learning training and inference, built around two encodings, E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2…

AI ComputeAI Hardware

Fine-tuning

Fine-tuning is the process of adapting a pretrained machine-learning model by continuing to optimize some or all of its parameters on data selected for a target task, domain, behavior, or population.

Deep LearningMachine Learning

Fisher information

Fisher information is a measure, defined by the statistician Ronald A. Fisher, of how much information an observable random variable carries about an unknown parameter of the probability distribution that…

Machine LearningMathematics

Focal loss

Focal loss is a loss function that reshapes standard cross-entropy loss by adding a (1 - p_t)^gamma modulating factor, which down-weights well-classified (easy) examples so that training concentrates on hard

Computer VisionDeep Learning

GRPO

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for fine-tuning large language models that eliminates the separate critic (value) network used by PPO

AI InferenceChinese AI

Genetic algorithm

A genetic algorithm (GA) is a search and optimization method that imitates natural selection: it keeps a population of candidate solutions encoded as strings, scores each one with a fitness function, and…

AlgorithmsMachine Learning

Gradient

In machine learning, the gradient is the vector of partial derivatives of a loss function with respect to every model parameter, and it points in the direction in which the loss increases most steeply.

Machine LearningMathematics

Gradient Accumulation

Gradient accumulation is a deep learning training technique that simulates a large batch size on limited GPU memory by summing the gradients from several small mini-batches (called micro-batches) and…

Deep LearningMachine Learning

Gradient Descent

Gradient descent is an iterative first-order method for minimizing a differentiable scalar objective. Starting from a point, it evaluates the local gradient and moves in the opposite direction.

Deep LearningMachine Learning

Gradient checkpointing

Gradient checkpointing, also called activation checkpointing, activation recomputation, or rematerialization, is a memory-saving technique for training deep neural networks that trades extra compute for much…

Deep Learning

Gradient clipping

Gradient clipping is a training technique that caps the magnitude of gradient values before they update model weights

Group normalization

Group normalization (GN, also written GroupNorm) is a normalization layer for deep neural networks that splits the channels of a feature map into groups and normalizes the features in each group, for each…

Computer VisionDeep Learning

Hinge Loss

Hinge loss is the margin-based loss function defined as max(0, 1 - y * f(x)), used to train support vector machines (SVMs) and other maximum-margin classifiers, where y in {+1, -1} is the true label and f(x)…

Machine Learning

Huber loss

Huber loss is a loss function for regression that is quadratic for small errors and linear for large ones, so it behaves like squared loss near zero and like absolute loss (L1 loss) in the tails.

Deep LearningMachine Learning

HuggingFace PEFT

PEFT (Parameter-Efficient Fine-Tuning) is an open-source Python library from Hugging Face that adapts large pretrained models to new tasks by training only a small set of added or selected parameters, often…

Developer ToolsOpen Source AI

InstructGPT

InstructGPT is a family of language models released by OpenAI in January 2022 that take the base GPT-3 and fine-tune it to follow user instructions more helpfully, truthfully, and with less toxic output, using…

AI AlignmentLarge Language Models

KTO

KTO (Kahneman-Tversky Optimization) is a method for aligning large language models with human feedback using only a binary signal of whether a model output is desirable or undesirable, rather than the paired…

AI AlignmentAI Inference

L0 Regularization

L0 regularization is a regularization technique in machine learning and statistics that penalizes the number of nonzero parameters in a model.

Machine Learning