AdamW
AdamW is a variant of the Adam optimizer that decouples weight decay from the gradient-based update rule, applying the decay directly to the weights instead of folding it into the loss as an L2 penalty.
Explore Training & Optimization through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of Training & Optimization.
Showing 1-60 of 80 articles
AdamW is a variant of the Adam optimizer that decouples weight decay from the gradient-based update rule, applying the decay directly to the weights instead of folding it into the loss as an L2 penalty.
Alternating least squares (ALS) is an iterative method for fitting a product of two or more factor matrices to data by holding all but one factor fixed, solving the resulting linear least squares problem for…
Bayesian optimization is a sequential, model-based strategy for finding the global optimum of expensive black-box functions in as few evaluations as possible.
Candidate sampling is a family of training-time optimization techniques used in machine learning to reduce the computational cost of models that must choose among a very large number of output classes.
Clipping is a family of techniques in machine learning that constrain numerical values to lie within a specified range or below a specified magnitude.
Convergence in machine learning is the point at which an iterative optimization algorithm reaches a stable solution, meaning the loss function stops decreasing meaningfully and further parameter updates yield…
A convex function is a real-valued function whose graph curves upward into a bowl or cup shape, so that the line segment (chord) connecting any two points on the graph lies on or above the graph itself.
Convex optimization is the branch of mathematical optimization that minimizes a convex function over a convex set, a problem class with one defining advantage: every local minimum is also a global minimum.
A convex set is a set of points in which the line segment connecting any two points of the set lies entirely within the set.
Curriculum learning is a training strategy for machine learning models in which training examples are presented in a meaningful, easy-to-hard order rather than at random
DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.
Distributionally robust optimization (DRO) is a framework for making decisions when the probability distribution of uncertain inputs is not known exactly.
DoRA (Weight-Decomposed Low-Rank Adaptation) is a parameter-efficient fine-tuning (PEFT) method for large neural networks introduced in February 2024 by researchers from NVIDIA, the Hong Kong University of…
Domain adaptation is the subfield of transfer learning that adapts a model trained on a labelled source domain so it performs well on a related but different target domain, where labels are scarce or absent
Dropout regularization is a regularization technique for neural networks that prevents overfitting by randomly setting a fraction of neuron activations to zero on each training step, forcing the network to…
Early stopping is a regularization technique that halts the training of an iterative machine learning model as soon as its performance on a held-out validation set stops improving
Elastic Net is a regularization method for linear regression that combines the L1 penalty associated with Lasso regression and the squared L2 penalty associated with ridge regression.
Empirical risk minimization (ERM) is the foundational principle of statistical learning theory: because the true risk (the expected loss over the unknown data distribution) cannot be computed
End-to-end learning is a way of building a machine learning system in which one model, usually a neural network, is trained to map raw input directly to the final output
Fine-tuning is the process of adapting a pretrained machine-learning model by continuing to optimize some or all of its parameters on data selected for a target task, domain, behavior, or population.
Fisher information is a measure, defined by the statistician Ronald A. Fisher, of how much information an observable random variable carries about an unknown parameter of the probability distribution that…
Focal loss is a loss function that reshapes standard cross-entropy loss by adding a (1 - p_t)^gamma modulating factor, which down-weights well-classified (easy) examples so that training concentrates on hard
GaLore (Gradient Low-Rank Projection) is a memory-efficient training strategy for large neural networks that projects each weight matrix's gradient into a low-rank subspace, computes the Adam-style optimizer…
A genetic algorithm (GA) is a search and optimization method that imitates natural selection: it keeps a population of candidate solutions encoded as strings, scores each one with a fitness function, and…
In machine learning, the gradient is the vector of partial derivatives of a loss function with respect to every model parameter, and it points in the direction in which the loss increases most steeply.
Gradient accumulation is a deep learning training technique that simulates a large batch size on limited GPU memory by summing the gradients from several small mini-batches (called micro-batches) and…
Gradient descent is an iterative first-order method for minimizing a differentiable scalar objective. Starting from a point, it evaluates the local gradient and moves in the opposite direction.
Hinge loss is the margin-based loss function defined as max(0, 1 - y * f(x)), used to train support vector machines (SVMs) and other maximum-margin classifiers, where y in {+1, -1} is the true label and f(x)…
Huber loss is a loss function for regression that is quadratic for small errors and linear for large ones, so it behaves like squared loss near zero and like absolute loss (L1 loss) in the tails.
A hyperparameter is an input that configures a machine learning algorithm or modeling pipeline, rather than an output fitted by that algorithm in one training run.
L0 regularization is a regularization technique in machine learning and statistics that penalizes the number of nonzero parameters in a model.
L1 loss is a regression loss function equal to the average of the absolute differences between predicted values and target values.
L1 regularization is a regularization technique in machine learning and statistics that prevents overfitting by adding the sum of the absolute values of a model's parameters as a penalty term to the loss…
L2 loss is the squared-error loss function: for a true value…
L2 regularization is a technique in machine learning and statistics that penalizes large weight values by adding the sum of squared parameters.
Label smoothing is a regularization technique for training classifiers in which the one-hot target label is replaced by a mixture of that label and a fixed distribution over classes, usually the uniform…
Lasso regression (an acronym for Least Absolute Shrinkage and Selection Operator) is a linear regression method, named and popularized by Robert Tibshirani in 1996, that adds an L1 penalty to the ordinary…
The learning rate is a hyperparameter that scales an update made by an iterative optimization algorithm. In machine learning, it is commonly written as η or α.
Low-Rank Adaptation, usually abbreviated LoRA, is a parameter-efficient fine-tuning method for adapting a pre-trained model.
Log loss is the negative log-likelihood of the predicted probabilities and the standard loss function for probabilistic classification: for binary labels it is computed as…
In machine learning, loss (sometimes called error) is a single non-negative number that measures how far a model's prediction is from the correct answer
A loss curve is a plot that shows the value of a loss function over the course of training a machine learning model.
A loss function assigns a numerical cost to a prediction, decision, or action after an outcome is observed.
The loss surface (also called the loss landscape, the error surface, or the objective function surface) is the geometric representation of a loss function as a function of the model's parameters.
The lottery ticket hypothesis is a conjecture about neural network training, proposed by Jonathan Frankle and Michael Carbin of MIT CSAIL, which holds that a dense
MXFP4 (Microscaling FP4) is a 4-bit block-scaled number format for quantization in machine learning, defined in the Open Compute Project (OCP) Microscaling Formats (MX) Specification v1.0 of September 2023.
Masked autoencoder (MAE) is a self-supervised learning method for vision transformers that masks roughly 75% of an input image's patches and trains a network to reconstruct the missing pixels from the small…
Minimax loss is a loss function rooted in game theory and decision theory that measures the worst-case performance of a strategy, algorithm, or model.
Multi-task learning (MTL) is a machine learning approach in which one model is trained on several tasks at the same time, so that what it learns for one task helps it learn the others.
A NaN trap (short for "Not a Number" trap) is a failure mode in machine learning training where arithmetic operations produce the special IEEE 754 value NaN, which then propagates through all subsequent…
Negative sampling is a training technique in machine learning in which a model learns to tell observed ("positive") examples apart from a small number of randomly drawn ("negative") examples
Next-token prediction is the training objective used by most modern language models: the model reads a prefix of tokenized text, outputs a probability distribution over which token comes next, and training…
ORPO (Odds Ratio Preference Optimization) is a preference alignment algorithm for large language models that merges supervised fine-tuning and preference alignment into a single training stage, eliminating the…
In machine learning, an objective (or objective function) is the scalar function that a learning algorithm optimizes during training: the single number whose gradients drive every parameter update.
Optimization (also called mathematical optimization) is the study of choosing the best value of a set of variables according to a numerical criterion, usually by minimizing or maximizing an objective function…
An optimizer in machine learning is an algorithm that iteratively adjusts a model's learnable parameters to minimize (or maximize) an objective function, commonly called a loss function
PROWL-1 is a training framework from the AI lab Odyssey in which a reinforcement learning agent is paid to break a world model.
Pipelining is a term used in two distinct senses within machine learning and artificial intelligence.
Policy gradient methods are a family of reinforcement learning algorithms that directly parameterise the agent's policy and optimise it by stochastic gradient ascent on the expected return.
A pre-trained model is a machine learning model that has already been trained on a large, general-purpose dataset and can then be reused, either as a fixed feature extractor or by fine-tuning