L1 Loss
L1 loss is a regression loss function equal to the average of the absolute differences between predicted values and target values.
Explore Training & Optimization through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of Training & Optimization.
Showing 61-120 of 169 articles
L1 loss is a regression loss function equal to the average of the absolute differences between predicted values and target values.
L1 regularization is a regularization technique in machine learning and statistics that prevents overfitting by adding the sum of the absolute values of a model's parameters as a penalty term to the loss…
L2 loss is the squared-error loss function: for a true value…
L2 regularization is a technique in machine learning and statistics that penalizes large weight values by adding the sum of squared parameters.
LIMA, short for "Less Is More for Alignment," is a 2023 research paper by Chunting Zhou and colleagues at Meta AI, Carnegie Mellon University, the University of Southern California, and Tel Aviv University…
LLaMA-Factory is an open-source unified framework for the efficient fine-tuning of large language models (LLMs) and vision-language models (VLMs).
Label smoothing is a regularization technique for training classifiers in which the one-hot target label is replaced by a mixture of that label and a fixed distribution over classes, usually the uniform…
Lasso regression (an acronym for Least Absolute Shrinkage and Selection Operator) is a linear regression method, named and popularized by Robert Tibshirani in 1996, that adds an L1 penalty to the ordinary…
The learning rate is a hyperparameter that scales an update made by an iterative optimization algorithm. In machine learning, it is commonly written as η or α.
Lion (EvoLved Sign Momentum) is a stochastic optimizer for training deep neural networks, introduced by researchers at Google in the February 2023 paper "Symbolic Discovery of Optimization Algorithms" by…
Low-Rank Adaptation, usually abbreviated LoRA, is a parameter-efficient fine-tuning method for adapting a pre-trained model.
LoftQ (short for LoRA-Fine-Tuning-aware Quantization) is a quantization and initialization framework for large language models that jointly quantizes a pre-trained backbone and initializes the attached…
Log loss is the negative log-likelihood of the predicted probabilities and the standard loss function for probabilistic classification: for binary labels it is computed as…
LongLoRA is a parameter-efficient fine-tuning technique that extends the context window of pre-trained large language models with substantially lower computation than full fine-tuning.
In machine learning, loss (sometimes called error) is a single non-negative number that measures how far a model's prediction is from the correct answer
A loss curve is a plot that shows the value of a loss function over the course of training a machine learning model.
A loss function assigns a numerical cost to a prediction, decision, or action after an outcome is observed.
The loss surface (also called the loss landscape, the error surface, or the objective function surface) is the geometric representation of a loss function as a function of the model's parameters.
The lottery ticket hypothesis is a conjecture about neural network training, proposed by Jonathan Frankle and Michael Carbin of MIT CSAIL, which holds that a dense
MXFP4 (Microscaling FP4) is a 4-bit block-scaled number format for quantization in machine learning, defined in the Open Compute Project (OCP) Microscaling Formats (MX) Specification v1.0 of September 2023.
Masked autoencoder (MAE) is a self-supervised learning method for vision transformers that masks roughly 75% of an input image's patches and trains a network to reconstruct the missing pixels from the small…
Megatron-LM is NVIDIA's open-source framework for training very large transformer language models across GPU clusters
Mini-batch stochastic gradient descent (often shortened to mini-batch SGD or MB-SGD) is the optimization algorithm used to train almost every modern neural network: it updates a model's parameters by…
Minimax loss is a loss function rooted in game theory and decision theory that measures the worst-case performance of a strategy, algorithm, or model.
Minitron is a family of compact language models from NVIDIA, together with the model-compression method used to build them: take one large, already pretrained large language model and shrink it into smaller…
Mistral Forge, branded by its developer simply as Forge, is an enterprise platform and engineering service from Mistral AI for building custom large language models and related multimodal models on an…
Model Predictive Control (MPC), also called receding horizon control (RHC), is a feedback control method that, at every sampling instant, solves a finite horizon optimal control problem using the current…
Momentum is an optimization technique that accelerates gradient descent by accumulating an exponentially weighted moving average of past gradients into a velocity vector and using that velocity, rather than…
Multi-task learning (MTL) is a machine learning approach in which one model is trained on several tasks at the same time, so that what it learns for one task helps it learn the others.
Muon (short for MomentUm Orthogonalized by Newton-Schulz) is a neural-network optimizer that updates the two-dimensional weight matrices of hidden layers by taking the momentum-based SGD update and…
NCCL, the NVIDIA Collective Communications Library, is a library of topology-aware communication primitives for NVIDIA GPU systems.
A NaN trap (short for "Not a Number" trap) is a failure mode in machine learning training where arithmetic operations produce the special IEEE 754 value NaN, which then propagates through all subsequent…
Negative sampling is a training technique in machine learning in which a model learns to tell observed ("positive") examples apart from a small number of randomly drawn ("negative") examples
Nemotron-Personas is a collection of open synthetic persona datasets published by NVIDIA on Hugging Face.
Next-token prediction is the training objective used by most modern language models: the model reads a prefix of tokenized text, outputs a probability distribution over which token comes next, and training…
NormalFloat 4-bit (NF4) is a 4-bit numerical data type for storing the weights of deep neural networks, introduced in the 2023 QLoRA paper by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer…
ORPO (Odds Ratio Preference Optimization) is a preference alignment algorithm for large language models that merges supervised fine-tuning and preference alignment into a single training stage, eliminating the…
In machine learning, an objective (or objective function) is the scalar function that a learning algorithm optimizes during training: the single number whose gradients drive every parameter update.
An objective function is the single scalar-valued quantity that an optimization algorithm tries to minimize or maximize during training.
Olmo-core is an open-source Python library for modeling and large-scale distributed training with PyTorch. The Allen Institute for AI (Ai2) develops and maintains it under the Apache 2.0 license.
Optimization (also called mathematical optimization) is the study of choosing the best value of a set of variables according to a numerical criterion, usually by minimizing or maximizing an objective function…
An optimizer in machine learning is an algorithm that iteratively adjusts a model's learnable parameters to minimize (or maximize) an objective function, commonly called a loss function
Parameter-efficient fine-tuning (PEFT) is a family of methods that adapt a pre-trained large language model to a downstream task by updating only a small fraction of its parameters
PROWL-1 is a training framework from the AI lab Odyssey in which a reinforcement learning agent is paid to break a world model.
The Parameter Server (PS) is a distributed system architecture for training large machine learning models in which one set of machines, the server nodes, holds the global model parameters as a partitioned…
A parameter update is the step in neural-network training where a model's trainable weights are adjusted using the gradient of the loss.
A partitioning strategy in distributed deep learning is the plan that decides how a model and its training data are split across multiple accelerators (typically GPUs or TPUs) so that the workload can be…
PiSSA (Principal Singular values and Singular vectors Adaptation) is a parameter-efficient fine-tuning method for large language models that initializes LoRA-style low-rank adapter matrices from the dominant…
Pipeline parallelism (often abbreviated PP) is a distributed training strategy that splits the layers of a deep neural network across multiple accelerator devices so that each device holds one contiguous block…
Pipelining is a term used in two distinct senses within machine learning and artificial intelligence.
Policy gradient methods are a family of reinforcement learning algorithms that directly parameterise the agent's policy and optimise it by stochastic gradient ascent on the expected return.
A pre-trained model is a machine learning model that has already been trained on a large, general-purpose dataset and can then be reused, either as a fixed feature extractor or by fine-tuning
Pre-training is a stage of machine learning in which a model learns parameters from a source dataset or source objective before those parameters are reused or adapted for a target use.
Proximal Policy Optimization (PPO) is an on-policy policy gradient reinforcement learning algorithm that stabilizes training by clipping the policy update so the new policy stays close ("proximal") to the old…
Pruning is a family of techniques used in machine learning and artificial intelligence to remove parts of a model or search space that are estimated to be unnecessary for accuracy or optimality.
QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…
RLOO (REINFORCE Leave-One-Out) is an online reinforcement learning algorithm for aligning large language models with reward signals such as those derived from human preferences.
Reinforcement Learning with Verifiable Rewards (RLVR) is a post-training paradigm for large language models in which the reward signal comes from a deterministic
RMSProp (Root Mean Square Propagation) is an adaptive learning-rate optimizer that divides each parameter's gradient by a running root-mean-square of that parameter's recent gradients
Regularization is any deliberate change to a learning problem or training algorithm intended to improve performance on data that were not used to fit the model.