Adafactor
Adafactor is an adaptive optimizer designed to reduce the memory used by second-moment estimates during neural-network training.
Training & Optimization
Explore Training & Optimization through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Deep Learning
Articles that also belong to these categories. Counts cover all of Training & Optimization.
All articles (145)Machine Learning (70)Deep Learning (41)Large Language Models (18)AI Infrastructure (9)Mathematics (9)AI Inference (8)Reinforcement Learning (8)AI Alignment (7)Developer Tools (7)Natural Language Processing (7)Open Source AI (7)MLOps (6)Statistics (6)NVIDIA (4)AI Hardware (3)Artificial Intelligence (3)Computer Vision (3)Reasoning Models (3)Algorithms (2)Chinese AI (2)Data & Datasets (2)Generative AI (2)Meta AI (2)Model Evaluation (2)Robotics (2)Transformer Models (2)AI Agents (1)AI Code Generation (1)AI Models (1)AI Research (1)AI Tools & Products (1)Enterprise AI (1)Google (1)Google DeepMind (1)Humanoid Robots (1)Microsoft (1)Mixture of Experts (1)Model Architecture (1)Neural Networks (1)OpenAI (1)Small Language Models (1)World Models (1)
Showing 1-41 of 41 articles
Clipping
Clipping is a family of techniques in machine learning that constrain numerical values to lie within a specified range or below a specified magnitude.
Cosine learning rate schedule
The cosine learning rate schedule, also called cosine annealing, is a learning rate decay strategy that lowers the optimizer step size from a peak value to a small minimum following a half period of a cosine…
Curriculum learning
Curriculum learning is a training strategy for machine learning models in which training examples are presented in a meaningful, easy-to-hard order rather than at random
DeepSpeed
DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.
Dropout
Dropout is a regularization technique for neural networks that randomly sets a fraction of neuron activations to zero during training
Dropout Regularization
Dropout regularization is a regularization technique for neural networks that prevents overfitting by randomly setting a fraction of neuron activations to zero on each training step, forcing the network to…
Early Stopping
Early stopping is a regularization technique that halts the training of an iterative machine learning model as soon as its performance on a held-out validation set stops improving
Fine Tuning
Fine-tuning is the process of adapting a pretrained machine-learning model by continuing to optimize some or all of its parameters on data selected for a target task, domain, behavior, or population.
Focal loss
Focal loss is a loss function that reshapes standard cross-entropy loss by adding a (1 - p_t)^gamma modulating factor, which down-weights well-classified (easy) examples so that training concentrates on hard
Fully Sharded Data Parallel (FSDP)
Fully Sharded Data Parallel (FSDP) is a distributed training technique implemented in PyTorch that shards a model's parameters, gradients, and optimizer states across data-parallel workers, allowing models…
Gradient Accumulation
Gradient accumulation is a deep learning training technique that simulates a large batch size on limited GPU memory by summing the gradients from several small mini-batches (called micro-batches) and…
Gradient Descent
Gradient descent is an iterative first-order method for minimizing a differentiable scalar objective. Starting from a point, it evaluates the local gradient and moves in the opposite direction.
Gradient checkpointing
Gradient checkpointing, also called activation checkpointing, activation recomputation, or rematerialization, is a memory-saving technique for training deep neural networks that trades extra compute for much…
Hyperparameter
A hyperparameter is an input that configures a machine learning algorithm or modeling pipeline, rather than an output fitted by that algorithm in one training run.
Learning Rate
The learning rate is a hyperparameter that scales an update made by an iterative optimization algorithm. In machine learning, it is commonly written as η or α.
LoRA (Low-Rank Adaptation)
Low-Rank Adaptation, usually abbreviated LoRA, is a parameter-efficient fine-tuning method for adapting a pre-trained model.
Loss Curve
A loss curve is a plot that shows the value of a loss function over the course of training a machine learning model.
Loss Function
A loss function assigns a numerical cost to a prediction, decision, or action after an outcome is observed.
Loss Surface
The loss surface (also called the loss landscape, the error surface, or the objective function surface) is the geometric representation of a loss function as a function of the model's parameters.
Multi-task learning
Multi-task learning (MTL) is a machine learning approach in which one model is trained on several tasks at the same time, so that what it learns for one task helps it learn the others.
Muon (optimizer)
Muon (short for MomentUm Orthogonalized by Newton-Schulz) is a neural-network optimizer that updates the two-dimensional weight matrices of hidden layers by taking the momentum-based SGD update and…
NaN Trap
A NaN trap (short for "Not a Number" trap) is a failure mode in machine learning training where arithmetic operations produce the special IEEE 754 value NaN, which then propagates through all subsequent…
Optimizer
An optimizer in machine learning is an algorithm that iteratively adjusts a model's learnable parameters to minimize (or maximize) an objective function, commonly called a loss function
Pre-Trained Model
A pre-trained model is a machine learning model that has already been trained on a large, general-purpose dataset and can then be reused, either as a fixed feature extractor or by fine-tuning
Pre-training
Pre-training is a stage of machine learning in which a model learns parameters from a source dataset or source objective before those parameters are reused or adapted for a target use.
QLoRA
QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…
RMSProp
RMSProp (Root Mean Square Propagation) is an adaptive learning-rate optimizer that divides each parameter's gradient by a running root-mean-square of that parameter's recent gradients
Regularization
Regularization is any deliberate change to a learning problem or training algorithm intended to improve performance on data that were not used to fit the model.
Representation learning
Representation learning is the area of machine learning concerned with getting a system to discover the features it needs for a task, instead of having people specify those features by hand.
SOAP (optimizer)
SOAP (ShampoO with Adam in the Preconditioner's eigenbasis) is a second-order optimization algorithm for training deep neural networks, introduced by Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai…
Schedule-Free optimizer
Schedule-Free is a family of optimization algorithms for deep learning and convex stochastic optimization that matches or exceeds the performance of tuned learning-rate schedules without specifying a horizon T…
Sequence Parallelism
Sequence parallelism (SP) is a family of distributed training techniques for transformer-based neural networks that partitions activations along the sequence (token) dimension across multiple accelerators…
Shampoo (optimizer)
Shampoo is a second-order stochastic optimization algorithm for training neural networks that maintains, for each parameter tensor
Sophia (optimizer)
Sophia (a backronym for Second-order Clipped Stochastic Optimization with Adaptive estimator) is a stochastic second-order optimization algorithm introduced in May 2023 by Hong Liu, Zhiyuan Li, David Hall…
Staged training
Staged training (also called multi-stage training or phased training) is a strategy in machine learning where a model is trained in two or more distinct sequential phases, each with its own objective, data…
Stochastic Gradient Descent (SGD)
Stochastic gradient descent (SGD) is a first-order optimization method that updates parameters using a gradient estimate computed from a randomly selected example or subset of examples.
Supervised fine-tuning
Supervised fine-tuning (SFT) is supervised training applied to a model that has already been trained.
ZeRO (Zero Redundancy Optimizer)
ZeRO (Zero Redundancy Optimizer) is a family of memory-optimization techniques for training large neural networks introduced by Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He at microsoft…
muP (Maximal Update Parametrization)
muP, short for Maximal Update Parametrization (often written muP, μP, or mu-P), is a parametrization scheme for deep neural networks in which a small set of optimization hyperparameters, most importantly the…
torch.compile
torch.compile is the just-in-time graph capture and compilation feature introduced in PyTorch 2.0, a release first announced at the PyTorch Conference on December 2, 2022 and shipped as a stable version on…