AdamW
AdamW is a variant of the Adam optimizer that decouples weight decay from the gradient-based update rule, applying the decay directly to the weights instead of folding it into the loss as an L2 penalty.
Training & Optimization
Explore Training & Optimization through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Machine Learning
Articles that also belong to these categories. Counts cover all of Training & Optimization.
All articles (145)Machine Learning (70)Deep Learning (41)Large Language Models (18)AI Infrastructure (9)Mathematics (9)AI Inference (8)Reinforcement Learning (8)AI Alignment (7)Developer Tools (7)Natural Language Processing (7)Open Source AI (7)MLOps (6)Statistics (6)NVIDIA (4)AI Hardware (3)Artificial Intelligence (3)Computer Vision (3)Reasoning Models (3)Algorithms (2)Chinese AI (2)Data & Datasets (2)Generative AI (2)Meta AI (2)Model Evaluation (2)Robotics (2)Transformer Models (2)AI Agents (1)AI Code Generation (1)AI Models (1)AI Research (1)AI Tools & Products (1)Enterprise AI (1)Google (1)Google DeepMind (1)Humanoid Robots (1)Microsoft (1)Mixture of Experts (1)Model Architecture (1)Neural Networks (1)OpenAI (1)Small Language Models (1)World Models (1)
Showing 1-60 of 70 articles
Bayesian Optimization
Bayesian optimization is a sequential, model-based strategy for finding the global optimum of expensive black-box functions in as few evaluations as possible.
Candidate Sampling
Candidate sampling is a family of training-time optimization techniques used in machine learning to reduce the computational cost of models that must choose among a very large number of output classes.
Clipping
Clipping is a family of techniques in machine learning that constrain numerical values to lie within a specified range or below a specified magnitude.
Convergence
Convergence in machine learning is the point at which an iterative optimization algorithm reaches a stable solution, meaning the loss function stops decreasing meaningfully and further parameter updates yield…
Convex Function
A convex function is a real-valued function whose graph curves upward into a bowl or cup shape, so that the line segment (chord) connecting any two points on the graph lies on or above the graph itself.
Convex Optimization
Convex optimization is the branch of mathematical optimization that minimizes a convex function over a convex set, a problem class with one defining advantage: every local minimum is also a global minimum.
Convex Set
A convex set is a set of points in which the line segment connecting any two points of the set lies entirely within the set .
Curriculum learning
Curriculum learning is a training strategy for machine learning models in which training examples are presented in a meaningful, easy-to-hard order rather than at random
DeepSpeed
DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.
Distributionally Robust Optimization
Distributionally robust optimization (DRO) is a framework for making decisions when the probability distribution of uncertain inputs is not known exactly.
DoRA (Weight-Decomposed Low-Rank Adaptation)
DoRA (Weight-Decomposed Low-Rank Adaptation) is a parameter-efficient fine-tuning (PEFT) method for large neural networks introduced in February 2024 by researchers from NVIDIA, the Hong Kong University of…
Domain adaptation
Domain adaptation is the subfield of transfer learning that adapts a model trained on a labelled source domain so it performs well on a related but different target domain, where labels are scarce or absent
Dropout Regularization
Dropout regularization is a regularization technique for neural networks that prevents overfitting by randomly setting a fraction of neuron activations to zero on each training step, forcing the network to…
Early Stopping
Early stopping is a regularization technique that halts the training of an iterative machine learning model as soon as its performance on a held-out validation set stops improving
Elastic Net
Elastic Net is a regularization method for linear regression that combines the L1 penalty associated with Lasso regression and the squared L2 penalty associated with ridge regression.
Empirical Risk Minimization
Empirical risk minimization (ERM) is the foundational principle of statistical learning theory: because the true risk (the expected loss over the unknown data distribution) cannot be computed
Fine Tuning
Fine-tuning is the process of adapting a pretrained machine-learning model by continuing to optimize some or all of its parameters on data selected for a target task, domain, behavior, or population.
Focal loss
Focal loss is a loss function that reshapes standard cross-entropy loss by adding a (1 - p_t)^gamma modulating factor, which down-weights well-classified (easy) examples so that training concentrates on hard
GaLore (Gradient Low-Rank Projection)
GaLore (Gradient Low-Rank Projection) is a memory-efficient training strategy for large neural networks that projects each weight matrix's gradient into a low-rank subspace, computes the Adam-style optimizer…
Gradient
In machine learning, the gradient is the vector of partial derivatives of a loss function with respect to every model parameter, and it points in the direction in which the loss increases most steeply.
Gradient Accumulation
Gradient accumulation is a deep learning training technique that simulates a large batch size on limited GPU memory by summing the gradients from several small mini-batches (called micro-batches) and…
Gradient Descent
Gradient descent is an iterative first-order method for minimizing a differentiable scalar objective. Starting from a point, it evaluates the local gradient and moves in the opposite direction.
Hinge Loss
Hinge loss is the margin-based loss function defined as max(0, 1 - y * f(x)), used to train support vector machines (SVMs) and other maximum-margin classifiers, where y in {+1, -1} is the true label and f(x)…
Hyperparameter
A hyperparameter is an input that configures a machine learning algorithm or modeling pipeline, rather than an output fitted by that algorithm in one training run.
L0 Regularization
L0 regularization is a regularization technique in machine learning and statistics that penalizes the number of nonzero parameters in a model, a quantity written $$\lVert \theta \rVert_0$$.
L1 Loss
L1 loss is a regression loss function equal to the average of the absolute differences between predicted values and target values, written as $$\frac{1}{n} \sum \lvert y_i - \hat{y}_i \rvert$$.
L1 Regularization
L1 regularization is a regularization technique in machine learning and statistics that prevents overfitting by adding the sum of the absolute values of a model's parameters as a penalty term to the loss…
L2 Loss
L2 loss is the squared-error loss function: for a true value $$y$$ and a predicted value $$\hat{y}$$, it is the squared difference $$(y - \hat{y})^2$$, and averaging it across a dataset gives the mean squared…
L2 Regularization
L2 regularization is a technique in machine learning and statistics that penalizes large weight values by adding the sum of squared parameters, scaled by a strength factor $$\lambda$$, to the loss function.
Lasso Regression
Lasso regression (an acronym for Least Absolute Shrinkage and Selection Operator) is a linear regression method, introduced by Robert Tibshirani in 1996, that adds an L1 penalty to the ordinary least squares…
Learning Rate
The learning rate is a hyperparameter that scales an update made by an iterative optimization algorithm. In machine learning, it is commonly written as η or α.
LoRA (Low-Rank Adaptation)
Low-Rank Adaptation, usually abbreviated LoRA, is a parameter-efficient fine-tuning method for adapting a pre-trained model.
Log Loss
Log loss is the negative log-likelihood of the predicted probabilities and the standard loss function for probabilistic classification: for binary labels it is computed as…
Loss
In machine learning, loss (sometimes called error) is a single non-negative number that measures how far a model's prediction is from the correct answer
Loss Curve
A loss curve is a plot that shows the value of a loss function over the course of training a machine learning model.
Loss Function
A loss function assigns a numerical cost to a prediction, decision, or action after an outcome is observed.
Loss Surface
The loss surface (also called the loss landscape, the error surface, or the objective function surface) is the geometric representation of a loss function as a function of the model's parameters.
Masked autoencoder (MAE)
Masked autoencoder (MAE) is a self-supervised learning method for vision transformers that masks roughly 75% of an input image's patches and trains a network to reconstruct the missing pixels from the small…
Minimax Loss
Minimax loss is a loss function rooted in game theory and decision theory that measures the worst-case performance of a strategy, algorithm, or model.
Multi-task learning
Multi-task learning (MTL) is a machine learning approach in which one model is trained on several tasks at the same time, so that what it learns for one task helps it learn the others.
Multi-token prediction
Multi-token prediction (often abbreviated MTP) is a language modeling training objective in which the model is trained to predict several future tokens at each context position rather than only the next token.
NaN Trap
A NaN trap (short for "Not a Number" trap) is a failure mode in machine learning training where arithmetic operations produce the special IEEE 754 value NaN, which then propagates through all subsequent…
Next-token prediction
Next-token prediction is the training objective used by most modern language models: the model reads a prefix of tokenized text, outputs a probability distribution over which token comes next, and training…
ORPO
ORPO (Odds Ratio Preference Optimization) is a preference alignment algorithm for large language models that merges supervised fine-tuning and preference alignment into a single training stage, eliminating the…
Objective
In machine learning, an objective (or objective function) is the scalar function that a learning algorithm optimizes during training: the single number whose gradients drive every parameter update.
Optimizer
An optimizer in machine learning is an algorithm that iteratively adjusts a model's learnable parameters to minimize (or maximize) an objective function, commonly called a loss function
PROWL-1
PROWL-1 is a training framework from the AI lab Odyssey in which a reinforcement learning agent is paid to break a world model.
Pipelining
Pipelining is a term used in two distinct senses within machine learning and artificial intelligence.
Policy gradient methods
Policy gradient methods are a family of reinforcement learning algorithms that directly parameterise the agent's policy and optimise it by stochastic gradient ascent on the expected return.
Pre-Trained Model
A pre-trained model is a machine learning model that has already been trained on a large, general-purpose dataset and can then be reused, either as a fixed feature extractor or by fine-tuning
Pre-training
Pre-training is a stage of machine learning in which a model learns parameters from a source dataset or source objective before those parameters are reused or adapted for a target use.
Proximal Policy Optimization (PPO)
Proximal Policy Optimization (PPO) is an on-policy policy gradient reinforcement learning algorithm that stabilizes training by clipping the policy update so the new policy stays close ("proximal") to the old…
Pruning
Pruning is a family of techniques used in machine learning and artificial intelligence to remove parts of a model or search space that are estimated to be unnecessary for accuracy or optimality.
Regularization
Regularization is any deliberate change to a learning problem or training algorithm intended to improve performance on data that were not used to fit the model.
Regularization Rate
The regularization rate (commonly denoted as $$\lambda$$ or alpha) is a hyperparameter that controls the strength of the penalty applied to a model's parameters during training.
Representation learning
Representation learning is the area of machine learning concerned with getting a system to discover the features it needs for a task, instead of having people specify those features by hand.
Ridge Regression
Ridge regression is a method of estimating the coefficients of a linear regression model by adding a penalty proportional to the sum of squared coefficients (the squared L2 norm) to the ordinary least squares…
Ridge Regularization
Ridge regularization, also known as L2 regularization or Tikhonov regularization, is a technique in statistics and machine learning that adds a squared L2-norm penalty to a model's loss function.
Shrinkage
Shrinkage in machine learning and statistics is a regularization technique that deliberately pulls model coefficient estimates toward zero, or toward some other fixed shrinkage target