Regularization Rate
"Regularization rate" is the terminology used in Google's Machine Learning Crash Course and Machine Learning Glossary
Explore Training & Optimization through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of Training & Optimization.
Showing 121-169 of 169 articles
"Regularization rate" is the terminology used in Google's Machine Learning Crash Course and Machine Learning Glossary
The reparameterization trick is a technique for computing low-variance gradients of an expectation with respect to the parameters of the distribution being sampled.
Representation learning is the area of machine learning concerned with getting a system to discover the features it needs for a task, instead of having people specify those features by hand.
Ridge regression is a method of estimating the coefficients of a linear regression model by adding a penalty proportional to the sum of squared coefficients (the squared L2 norm) to the ordinary least squares…
Ridge regularization, also known as L2 regularization or Tikhonov regularization, is a technique in statistics and machine learning that adds a squared L2-norm penalty to a model's loss function.
Ring Attention, formally Ring Attention with Blockwise Transformers, is a distributed algorithm for computing the self-attention operation of transformer neural networks across a ring of compute devices
SOAP (ShampoO with Adam in the Preconditioner's eigenbasis) is a second-order optimization algorithm for training deep neural networks, introduced by Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai…
SPIN (Self-Play fIne-tuNing) is a post-training method for large language models introduced by researchers at the University of California, Los Angeles (UCLA) in January 2024.
Schedule-Free is a family of optimization algorithms for deep learning and convex stochastic optimization that matches or exceeds the performance of tuned learning-rate schedules without specifying a horizon T…
Self-Instruct is a semi-automated framework for aligning a pretrained large language model with instruction-following behavior by bootstrapping its training data from the model itself, introduced in December…
Self-Rewarding Language Models (SRLM) is an iterative alignment method in which a single large language model alternately plays the role of policy (generating candidate responses to user prompts) and reward…
Self-Taught Evaluator is a method for training a strong LLM-as-a-judge without any human preference annotations, using synthetic training data and an iterative self-improvement loop.
Sequence parallelism (SP) is a family of distributed training techniques for transformer-based neural networks that partitions activations along the sequence (token) dimension across multiple accelerators…
Shampoo is a second-order stochastic optimization algorithm for training neural networks that maintains, for each parameter tensor
Shrinkage in machine learning and statistics is a regularization technique that deliberately pulls model coefficient estimates toward zero, or toward some other fixed shrinkage target
SimPO (Simple Preference Optimization) is a reference-free offline preference learning algorithm for aligning large language models with human preferences.
Simulated annealing is a probabilistic method for finding a good approximate solution to a global optimization problem, especially one with many local optima that trap ordinary hill-climbing methods.
SkyRL is an open-source reinforcement learning (RL) library for post-training large language models and multi-turn agents.
Sophia (a backronym for Second-order Clipped Stochastic Optimization with Adaptive estimator) is a stochastic second-order optimization algorithm introduced in May 2023 by Hong Liu, Zhiyuan Li, David Hall…
Squared hinge loss (also called L2 hinge loss or L2-loss) is a loss function used in machine learning for classification tasks, most commonly in support vector machines (SVMs).
Squared loss, also called quadratic loss, L2 loss, or squared error loss, is a loss function that penalizes a prediction by the square of its error: for a true value $y$ and a prediction $\hat{y}$
Staged training (also called multi-stage training or phased training) is a strategy in machine learning where a model is trained in two or more distinct sequential phases, each with its own objective, data…
In machine learning, a step is one discrete update in an iterative process, most often a single update of a model's parameters computed from one mini-batch of training data (also called a training step…
In machine learning, the step size…
Stochastic gradient descent (SGD) is a first-order optimization method that updates parameters using a gradient estimate computed from a randomly selected example or subset of examples.
Supervised fine-tuning (SFT) is supervised training applied to a model that has already been trained.
Swarm intelligence (SI) is a branch of artificial intelligence concerned with the collective behavior of decentralized, self-organized systems
A Tensor Core is a specialized execution unit inside NVIDIA GPUs that computes a small matrix multiplication and accumulation, D = A x B + C
Tensor parallelism (TP) is a distributed training technique that splits the individual weight matrices of a neural network layer across multiple devices, so that each device computes a partial result that is…
A termination condition, also called a stopping criterion, convergence criterion, or halting condition, is a rule that decides when an iterative algorithm should stop running.
Together Post-Training is a managed service from Together AI for customizing open-weight large language models with reinforcement learning (RL) and supervised fine-tuning (SFT).
TorchTitan (styled torchtitan in its repository) is an open-source, PyTorch-native platform for training large generative AI models
Training compute is the total number of floating-point operations (FLOP) performed to train a machine learning model, normally counted over the forward and backward passes of the final training run.
In machine learning, training loss is the value of the loss function computed on the training data during model training, and it is the exact quantity that the optimization algorithm minimizes at each step.
A training run is a single, deliberate instance of training a neural network from scratch (or from a prior checkpoint) on a specified dataset, with a fixed compute budget, hardware allocation, and time horizon.
Trust Region Policy Optimization (TRPO) is a policy gradient algorithm for reinforcement learning that improves a stochastic policy by maximizing a surrogate objective while keeping the average KL divergence…
Unsloth is an open-source Python library that fine-tunes large language models up to two times faster while using up to 70 percent less GPU memory and, in its own words, with "no accuracy loss".
Vector-based Random Matrix Adaptation (VeRA) is a parameter-efficient fine-tuning (PEFT) method for adapting large pretrained neural networks to downstream tasks while training only a tiny fraction of the…
WRAP (Web Rephrase Augmented Pre-training) is a synthetic-data pre-training method introduced in the paper "Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling," posted to arXiv on 29…
Wasserstein loss is a loss function for training generative models that measures the distance between two probability distributions as the Wasserstein-1 distance
Weight decay is a regularization technique used in training neural networks that shrinks every parameter toward zero by a small fraction on each update step
Weight initialization is the choice of starting values for the trainable parameters of a neural network before training begins.
Weight-Space Programming is an approach in which a general-purpose large language model reads, interprets, edits or constructs another neural network's numerical weights in response to a natural-language goal
Whole-body control (WBC) is a class of robotics control techniques that coordinates all of a robot's degrees of freedom at once to achieve multiple prioritized tasks, such as balancing, reaching, and…
ZeRO (Zero Redundancy Optimizer) is a family of memory-optimization techniques for training large neural networks introduced by Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He at microsoft…
Zephon is an open-source data loader for training foundation models on text and multimodal data, developed by DatologyAI and released under the Apache License 2.0 in October 2026.
muP, short for Maximal Update Parametrization (often written muP, μP, or mu-P), is a parametrization scheme for deep neural networks in which a small set of optimization hyperparameters, most importantly the…
rsLoRA (rank-stabilized LoRA) is a one-line modification to the low-rank adaptation fine-tuning method for large neural networks.
torch.compile is the just-in-time graph capture and compilation feature introduced in PyTorch 2.0, a release first announced at the PyTorch Conference on December 2, 2022 and shipped as a stable version on…