Training & Optimization

Explore Training & Optimization through related topics and the articles other pages reference most.

Explore articles

Browse subtopics (47)

Articles that also belong to these categories. Counts cover all of Training & Optimization.

Showing 121-169 of 169 articles

Ridge Regression

Ridge regression is a method of estimating the coefficients of a linear regression model by adding a penalty proportional to the sum of squared coefficients (the squared L2 norm) to the ordinary least squares…

Machine Learning

Ridge Regularization

Ridge regularization, also known as L2 regularization or Tikhonov regularization, is a technique in statistics and machine learning that adds a squared L2-norm penalty to a model's loss function.

Machine LearningStatistics

Ring Attention

Ring Attention, formally Ring Attention with Blockwise Transformers, is a distributed algorithm for computing the self-attention operation of transformer neural networks across a ring of compute devices

Transformer Models

SOAP (optimizer)

SOAP (ShampoO with Adam in the Preconditioner's eigenbasis) is a second-order optimization algorithm for training deep neural networks, introduced by Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai…

Deep Learning

Schedule-Free optimizer

Schedule-Free is a family of optimization algorithms for deep learning and convex stochastic optimization that matches or exceeds the performance of tuned learning-rate schedules without specifying a horizon T…

Deep Learning

Self-Instruct

Self-Instruct is a semi-automated framework for aligning a pretrained large language model with instruction-following behavior by bootstrapping its training data from the model itself, introduced in December…

Data & DatasetsLarge Language Models

Self-Taught Evaluator

Self-Taught Evaluator is a method for training a strong LLM-as-a-judge without any human preference annotations, using synthetic training data and an iterative self-improvement loop.

Meta AIModel Evaluation

Sequence Parallelism

Sequence parallelism (SP) is a family of distributed training techniques for transformer-based neural networks that partitions activations along the sequence (token) dimension across multiple accelerators…

AI InfrastructureDeep Learning

Shampoo (optimizer)

Shampoo is a second-order stochastic optimization algorithm for training neural networks that maintains, for each parameter tensor

Deep Learning

Shrinkage

Shrinkage in machine learning and statistics is a regularization technique that deliberately pulls model coefficient estimates toward zero, or toward some other fixed shrinkage target

Machine Learning

SimPO

SimPO (Simple Preference Optimization) is a reference-free offline preference learning algorithm for aligning large language models with human preferences.

AI AlignmentLarge Language Models

Simulated annealing

Simulated annealing is a probabilistic method for finding a good approximate solution to a global optimization problem, especially one with many local optima that trap ordinary hill-climbing methods.

AlgorithmsMachine Learning

Sophia (optimizer)

Sophia (a backronym for Second-order Clipped Stochastic Optimization with Adaptive estimator) is a stochastic second-order optimization algorithm introduced in May 2023 by Hong Liu, Zhiyuan Li, David Hall…

Deep Learning

Squared Hinge Loss

Squared hinge loss (also called L2 hinge loss or L2-loss) is a loss function used in machine learning for classification tasks, most commonly in support vector machines (SVMs).

Machine Learning

Squared Loss

Squared loss, also called quadratic loss, L2 loss, or squared error loss, is a loss function that penalizes a prediction by the square of its error: for a true value $y$ and a prediction $\hat{y}$

Machine LearningStatistics

Staged training

Staged training (also called multi-stage training or phased training) is a strategy in machine learning where a model is trained in two or more distinct sequential phases, each with its own objective, data…

Deep Learning

Step

In machine learning, a step is one discrete update in an iterative process, most often a single update of a model's parameters computed from one mini-batch of training data (also called a training step…

Step size

In machine learning, the step size…

Tensor Core

A Tensor Core is a specialized execution unit inside NVIDIA GPUs that computes a small matrix multiplication and accumulation, D = A x B + C

AI ComputeAI Hardware

Termination condition

A termination condition, also called a stopping criterion, convergence criterion, or halting condition, is a rule that decides when an iterative algorithm should stop running.

Machine Learning

TorchTitan

TorchTitan (styled torchtitan in its repository) is an open-source, PyTorch-native platform for training large generative AI models

Developer ToolsMeta AI

Training compute

Training compute is the total number of floating-point operations (FLOP) performed to train a machine learning model, normally counted over the forward and backward passes of the final training run.

AI ComputeMachine Learning

Training loss

In machine learning, training loss is the value of the loss function computed on the training data during model training, and it is the exact quantity that the optimization algorithm minimizes at each step.

Training run

A training run is a single, deliberate instance of training a neural network from scratch (or from a prior checkpoint) on a specified dataset, with a fixed compute budget, hardware allocation, and time horizon.

Machine Learning

Unsloth

Unsloth is an open-source Python library that fine-tunes large language models up to two times faster while using up to 70 percent less GPU memory and, in its own words, with "no accuracy loss".

Developer ToolsOpen Source AI

Weight Decay

Weight decay is a regularization technique used in training neural networks that shrinks every parameter toward zero by a small fraction on each update step

Machine Learning

Weight-Space Programming

Weight-Space Programming is an approach in which a general-purpose large language model reads, interprets, edits or constructs another neural network's numerical weights in response to a natural-language goal

AI ResearchInterpretability

Whole-body control

Whole-body control (WBC) is a class of robotics control techniques that coordinates all of a robot's degrees of freedom at once to achieve multiple prioritized tasks, such as balancing, reaching, and…

Humanoid RobotsRobotics

Zephon

Zephon is an open-source data loader for training foundation models on text and multimodal data, developed by DatologyAI and released under the Apache License 2.0 in October 2026.

Data & DatasetsDeveloper Tools

muP (Maximal Update Parametrization)

muP, short for Maximal Update Parametrization (often written muP, μP, or mu-P), is a parametrization scheme for deep neural networks in which a small set of optimization hyperparameters, most importantly the…

Deep Learning

torch.compile

torch.compile is the just-in-time graph capture and compilation feature introduced in PyTorch 2.0, a release first announced at the PyTorch Conference on December 2, 2022 and shipped as a stable version on…

Deep LearningDeveloper Tools