Training & Optimization

Sophia (optimizer)

Sophia (a backronym for Second-order Clipped Stochastic Optimization with Adaptive estimator) is a stochastic second-order optimization algorithm introduced in May 2023 by Hong Liu, Zhiyuan Li, David Hall…

Deep Learning

Squared Hinge Loss

Squared hinge loss (also called L2 hinge loss or L2-loss) is a loss function used in machine learning for classification tasks, most commonly in support vector machines (SVMs).

Machine Learning

Squared Loss

Squared loss, also called quadratic loss, L2 loss, or squared error loss, is a loss function that penalizes a prediction by the square of its error: for a true value $y$ and a prediction $\hat{y}$

Machine LearningStatistics

Staged training

Staged training (also called multi-stage training or phased training) is a strategy in machine learning where a model is trained in two or more distinct sequential phases, each with its own objective, data…

Deep Learning

Step

In machine learning, a step is one discrete update in an iterative process, most often a single update of a model's parameters computed from one mini-batch of training data (also called a training step…

Step size

In machine learning, the step size (also called the learning rate, usually written as the Greek letter $$\eta$$ or $$\alpha$$) is the scalar that controls how far the parameters of a model move on each update…

Stochastic Gradient Descent (SGD)

Stochastic gradient descent (SGD) is a first-order optimization method that updates parameters using a gradient estimate computed from a randomly selected example or subset of examples.

Deep LearningMachine Learning

Supervised fine-tuning

Supervised fine-tuning (SFT) is supervised training applied to a model that has already been trained.

Artificial IntelligenceDeep Learning

Swarm intelligence

Swarm intelligence (SI) is a branch of artificial intelligence concerned with the collective behavior of decentralized, self-organized systems

AI AgentsArtificial Intelligence

Tensor Core

A Tensor Core is a specialized execution unit inside NVIDIA GPUs that computes a small matrix multiplication and accumulation, D = A x B + C

AI HardwareAI Infrastructure

Tensor Parallelism

Tensor parallelism (TP) is a distributed training technique that splits the individual weight matrices of a neural network layer across multiple devices, so that each device computes a partial result that is…

AI InfrastructureLarge Language Models

Termination condition

A termination condition, also called a stopping criterion, convergence criterion, or halting condition, is a rule that decides when an iterative algorithm should stop running.

Machine Learning

Test-Time Training (TTT)

Test-Time Training (TTT) is a family of machine learning techniques in which a model updates a subset of its own parameters at inference time

AI InferenceMachine Learning

Training loss

In machine learning, training loss is the value of the loss function computed on the training data during model training, and it is the exact quantity that the optimization algorithm minimizes at each step.

Training run

A training run is a single, deliberate instance of training a neural network from scratch (or from a prior checkpoint) on a specified dataset, with a fixed compute budget, hardware allocation, and time horizon.

Machine Learning

Unsloth

Unsloth is an open-source Python library that fine-tunes large language models up to two times faster while using up to 70 percent less GPU memory and, in its own words, with "no accuracy loss".

Developer ToolsOpen Source AI

VeRA (Vector-based Random Matrix Adaptation)

Vector-based Random Matrix Adaptation (VeRA) is a parameter-efficient fine-tuning (PEFT) method for adapting large pretrained neural networks to downstream tasks while training only a tiny fraction of the…

Large Language Models

WRAP (Web Rephrase Augmented Pre-training)

WRAP (Web Rephrase Augmented Pre-training) is a synthetic-data pre-training method introduced in the paper "Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling," posted to arXiv on 29…

Data & DatasetsLarge Language Models

Wasserstein Loss

Wasserstein loss is a loss function for training generative models that measures the distance between two probability distributions as the Wasserstein-1 distance

Generative AIMachine Learning

Weight Decay

Weight decay is a regularization technique used in training neural networks that shrinks every parameter toward zero by a small fraction on each update step

Machine Learning

Whole-body control

Whole-body control (WBC) is a class of robotics control techniques that coordinates all of a robot's degrees of freedom at once to achieve multiple prioritized tasks, such as balancing, reaching, and…

Humanoid RobotsRobotics

ZeRO (Zero Redundancy Optimizer)

ZeRO (Zero Redundancy Optimizer) is a family of memory-optimization techniques for training large neural networks introduced by Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He at microsoft…

Deep LearningMicrosoft

muP (Maximal Update Parametrization)

muP, short for Maximal Update Parametrization (often written muP, μP, or mu-P), is a parametrization scheme for deep neural networks in which a small set of optimization hyperparameters, most importantly the…

Deep Learning

rsLoRA (Rank-Stabilized LoRA)

rsLoRA (rank-stabilized LoRA) is a one-line modification to the low-rank adaptation fine-tuning method for large neural networks.

Large Language Models

torch.compile

torch.compile is the just-in-time graph capture and compilation feature introduced in PyTorch 2.0, a release first announced at the PyTorch Conference on December 2, 2022 and shipped as a stable version on…

Deep LearningDeveloper Tools