AdaGrad
AdaGrad (short for Adaptive Gradient Algorithm) is an optimizer for gradient descent-based machine learning that gives every parameter its own learning rate
Training & Optimization
Explore Training & Optimization through related topics and the articles other pages reference most.
Most referenced in this topic
Ranked by links from other AI Wiki pages.
Explore articles
Browse subtopics (42)
Articles that also belong to these categories. Counts cover all of Training & Optimization.
All articles (145)Machine Learning (70)Deep Learning (41)Large Language Models (18)AI Infrastructure (9)Mathematics (9)AI Inference (8)Reinforcement Learning (8)AI Alignment (7)Developer Tools (7)Natural Language Processing (7)Open Source AI (7)MLOps (6)Statistics (6)NVIDIA (4)AI Hardware (3)Artificial Intelligence (3)Computer Vision (3)Reasoning Models (3)Algorithms (2)Chinese AI (2)Data & Datasets (2)Generative AI (2)Meta AI (2)Model Evaluation (2)Robotics (2)Transformer Models (2)AI Agents (1)AI Code Generation (1)AI Models (1)AI Research (1)AI Tools & Products (1)Enterprise AI (1)Google (1)Google DeepMind (1)Humanoid Robots (1)Microsoft (1)Mixture of Experts (1)Model Architecture (1)Neural Networks (1)OpenAI (1)Small Language Models (1)World Models (1)
Showing 1-60 of 145 articles
Adafactor
Adafactor is an adaptive optimizer designed to reduce the memory used by second-moment estimates during neural-network training.
Adam optimizer
The Adam optimizer (short for Adaptive Moment Estimation) is an algorithm for first-order gradient descent-based optimization of stochastic objective functions
AdamW
AdamW is a variant of the Adam optimizer that decouples weight decay from the gradient-based update rule, applying the decay directly to the weights instead of folding it into the loss as an L2 penalty.
AutoML (Automated Machine Learning)
AutoML (Automated Machine Learning) is the automation of the end-to-end pipeline of applying machine learning to real-world data, replacing manual trial and error with a formal search over data preprocessing…
Axolotl
Axolotl is a free and open source framework for fine-tuning and post-training large language models, written in Python and driven entirely by a single YAML configuration file.
Bayesian Optimization
Bayesian optimization is a sequential, model-based strategy for finding the global optimum of expensive black-box functions in as few evaluations as possible.
Candidate Sampling
Candidate sampling is a family of training-time optimization techniques used in machine learning to reduce the computational cost of models that must choose among a very large number of output classes.
Clipping
Clipping is a family of techniques in machine learning that constrain numerical values to lie within a specified range or below a specified magnitude.
Context Parallelism
Context Parallelism (CP) is a distributed training strategy that partitions the input sequence dimension of a transformer across multiple accelerators and uses ring-style point-to-point communication to…
Convergence
Convergence in machine learning is the point at which an iterative optimization algorithm reaches a stable solution, meaning the loss function stops decreasing meaningfully and further parameter updates yield…
Convex Function
A convex function is a real-valued function whose graph curves upward into a bowl or cup shape, so that the line segment (chord) connecting any two points on the graph lies on or above the graph itself.
Convex Optimization
Convex optimization is the branch of mathematical optimization that minimizes a convex function over a convex set, a problem class with one defining advantage: every local minimum is also a global minimum.
Convex Set
A convex set is a set of points in which the line segment connecting any two points of the set lies entirely within the set .
Cosine learning rate schedule
The cosine learning rate schedule, also called cosine annealing, is a learning rate decay strategy that lowers the optimizer step size from a peak value to a small minimum following a half period of a cosine…
Cost
In machine learning, cost is the scalar number that summarizes how badly a model is performing on a chunk of data.
Curriculum learning
Curriculum learning is a training strategy for machine learning models in which training examples are presented in a meaningful, easy-to-hard order rather than at random
DPO
DPO (Direct Preference Optimization) is an alignment technique for large language models that directly optimizes a language model policy from human preference data, without training a separate reward model or…
DeepSeek-R1-Distill
DeepSeek-R1-Distill is a family of six open-weight reasoning language models released by DeepSeek on January 20, 2025, alongside the flagship DeepSeek-R1 reasoning model.
DeepSpeed
DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.
DiLoCo
DiLoCo (Distributed Low-Communication training) is a distributed optimization algorithm for neural networks introduced by Google DeepMind in November 2023 to train large language models across loosely…
Distributed training
Distributed training is the practice of training a single machine learning model using many compute devices in parallel, splitting the data, the model, or both across GPUs, TPUs, or other accelerators that…
Distributionally Robust Optimization
Distributionally robust optimization (DRO) is a framework for making decisions when the probability distribution of uncertain inputs is not known exactly.
DoRA (Weight-Decomposed Low-Rank Adaptation)
DoRA (Weight-Decomposed Low-Rank Adaptation) is a parameter-efficient fine-tuning (PEFT) method for large neural networks introduced in February 2024 by researchers from NVIDIA, the Hong Kong University of…
Domain adaptation
Domain adaptation is the subfield of transfer learning that adapts a model trained on a labelled source domain so it performs well on a related but different target domain, where labels are scarce or absent
Dropout
Dropout is a regularization technique for neural networks that randomly sets a fraction of neuron activations to zero during training
Dropout Regularization
Dropout regularization is a regularization technique for neural networks that prevents overfitting by randomly setting a fraction of neuron activations to zero on each training step, forcing the network to…
Early Stopping
Early stopping is a regularization technique that halts the training of an iterative machine learning model as soon as its performance on a held-out validation set stops improving
Elastic Net
Elastic Net is a regularization method for linear regression that combines the L1 penalty associated with Lasso regression and the squared L2 penalty associated with ridge regression.
Empirical Risk Minimization
Empirical risk minimization (ERM) is the foundational principle of statistical learning theory: because the true risk (the expected loss over the unknown data distribution) cannot be computed
Expert Parallelism
Expert Parallelism (EP) is a model-parallelism strategy specific to Mixture of Experts (MoE) neural networks in which the individual expert sub-networks (typically feed-forward blocks) are sharded across…
FP4 (4-bit floating point)
FP4 (4-bit floating point) is a numerical format that stores a real number in just 4 bits, the smallest floating-point type in mainstream use for deep learning.
Fill-in-the-middle
Fill-in-the-middle (FIM) is a training objective and inference technique that lets an autoregressive language model generate text for a gap in the middle of a document, conditioned on both the text before the…
Fine Tuning
Fine-tuning is the process of adapting a pretrained machine-learning model by continuing to optimize some or all of its parameters on data selected for a target task, domain, behavior, or population.
Focal loss
Focal loss is a loss function that reshapes standard cross-entropy loss by adding a (1 - p_t)^gamma modulating factor, which down-weights well-classified (easy) examples so that training concentrates on hard
Fully Sharded Data Parallel (FSDP)
Fully Sharded Data Parallel (FSDP) is a distributed training technique implemented in PyTorch that shards a model's parameters, gradients, and optimizer states across data-parallel workers, allowing models…
GRPO
Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for fine-tuning large language models that eliminates the separate critic (value) network used by PPO
GaLore (Gradient Low-Rank Projection)
GaLore (Gradient Low-Rank Projection) is a memory-efficient training strategy for large neural networks that projects each weight matrix's gradient into a low-rank subspace, computes the Adam-style optimizer…
Gradient
In machine learning, the gradient is the vector of partial derivatives of a loss function with respect to every model parameter, and it points in the direction in which the loss increases most steeply.
Gradient Accumulation
Gradient accumulation is a deep learning training technique that simulates a large batch size on limited GPU memory by summing the gradients from several small mini-batches (called micro-batches) and…
Gradient Descent
Gradient descent is an iterative first-order method for minimizing a differentiable scalar objective. Starting from a point, it evaluates the local gradient and moves in the opposite direction.
Gradient checkpointing
Gradient checkpointing, also called activation checkpointing, activation recomputation, or rematerialization, is a memory-saving technique for training deep neural networks that trades extra compute for much…
Gradient clipping
Gradient clipping is a training technique that caps the magnitude of gradient values before they update model weights
Hinge Loss
Hinge loss is the margin-based loss function defined as max(0, 1 - y * f(x)), used to train support vector machines (SVMs) and other maximum-margin classifiers, where y in {+1, -1} is the true label and f(x)…
HuggingFace PEFT
PEFT (Parameter-Efficient Fine-Tuning) is an open-source Python library from Hugging Face that adapts large pretrained models to new tasks by training only a small set of added or selected parameters, often…
HuggingFace TRL
TRL (Transformer Reinforcement Learning, now stylized as Transformers Reinforcement Learning) is an open-source Python library maintained by Hugging Face for post-training large language models with…
Hyperparameter
A hyperparameter is an input that configures a machine learning algorithm or modeling pipeline, rather than an output fitted by that algorithm in one training run.
InstructGPT
InstructGPT is a family of language models released by OpenAI in January 2022 that take the base GPT-3 and fine-tune it to follow user instructions more helpfully, truthfully, and with less toxic output, using…
KTO
KTO (Kahneman-Tversky Optimization) is a method for aligning large language models with human feedback using only a binary signal of whether a model output is desirable or undesirable, rather than the paired…
L0 Regularization
L0 regularization is a regularization technique in machine learning and statistics that penalizes the number of nonzero parameters in a model, a quantity written $$\lVert \theta \rVert_0$$.
L1 Loss
L1 loss is a regression loss function equal to the average of the absolute differences between predicted values and target values, written as $$\frac{1}{n} \sum \lvert y_i - \hat{y}_i \rvert$$.
L1 Regularization
L1 regularization is a regularization technique in machine learning and statistics that prevents overfitting by adding the sum of the absolute values of a model's parameters as a penalty term to the loss…
L2 Loss
L2 loss is the squared-error loss function: for a true value $$y$$ and a predicted value $$\hat{y}$$, it is the squared difference $$(y - \hat{y})^2$$, and averaging it across a dataset gives the mean squared…
L2 Regularization
L2 regularization is a technique in machine learning and statistics that penalizes large weight values by adding the sum of squared parameters, scaled by a strength factor $$\lambda$$, to the loss function.
LIMA (Less Is More for Alignment)
LIMA, short for "Less Is More for Alignment," is a 2023 research paper by Chunting Zhou and colleagues at Meta AI, Carnegie Mellon University, the University of Southern California, and Tel Aviv University…
LLaMA-Factory
LLaMA-Factory is an open-source unified framework for the efficient fine-tuning of large language models (LLMs) and vision-language models (VLMs).
Lasso Regression
Lasso regression (an acronym for Least Absolute Shrinkage and Selection Operator) is a linear regression method, introduced by Robert Tibshirani in 1996, that adds an L1 penalty to the ordinary least squares…
Learning Rate
The learning rate is a hyperparameter that scales an update made by an iterative optimization algorithm. In machine learning, it is commonly written as η or α.
Lion (optimizer)
Lion (EvoLved Sign Momentum) is a stochastic optimizer for training deep neural networks, introduced by researchers at Google in the February 2023 paper "Symbolic Discovery of Optimization Algorithms" by…
LoRA (Low-Rank Adaptation)
Low-Rank Adaptation, usually abbreviated LoRA, is a parameter-efficient fine-tuning method for adapting a pre-trained model.