Training & Optimization

Explore Training & Optimization through related topics and the articles other pages reference most.

Explore articles

Browse subtopics (47)

Articles that also belong to these categories. Counts cover all of Training & Optimization.

Showing 61-120 of 169 articles

L1 Regularization

L1 regularization is a regularization technique in machine learning and statistics that prevents overfitting by adding the sum of the absolute values of a model's parameters as a penalty term to the loss…

Machine Learning

L2 Regularization

L2 regularization is a technique in machine learning and statistics that penalizes large weight values by adding the sum of squared parameters.

Machine Learning

LIMA (Less Is More for Alignment)

LIMA, short for "Less Is More for Alignment," is a 2023 research paper by Chunting Zhou and colleagues at Meta AI, Carnegie Mellon University, the University of Southern California, and Tel Aviv University…

AI ResearchMeta AI

Label smoothing

Label smoothing is a regularization technique for training classifiers in which the one-hot target label is replaced by a mixture of that label and a fixed distribution over classes, usually the uniform…

Deep LearningMachine Learning

Lasso Regression

Lasso regression (an acronym for Least Absolute Shrinkage and Selection Operator) is a linear regression method, named and popularized by Robert Tibshirani in 1996, that adds an L1 penalty to the ordinary…

Machine Learning

Lion (optimizer)

Lion (EvoLved Sign Momentum) is a stochastic optimizer for training deep neural networks, introduced by researchers at Google in the February 2023 paper "Symbolic Discovery of Optimization Algorithms" by…

AlgorithmsGoogle

LoftQ

LoftQ (short for LoRA-Fine-Tuning-aware Quantization) is a quantization and initialization framework for large language models that jointly quantizes a pre-trained backbone and initializes the attached…

Large Language Models

Log Loss

Log loss is the negative log-likelihood of the predicted probabilities and the standard loss function for probabilistic classification: for binary labels it is computed as…

Machine LearningMathematics

LongLoRA

LongLoRA is a parameter-efficient fine-tuning technique that extends the context window of pre-trained large language models with substantially lower computation than full fine-tuning.

Large Language Models

Loss

In machine learning, loss (sometimes called error) is a single non-negative number that measures how far a model's prediction is from the correct answer

Machine Learning

Loss Surface

The loss surface (also called the loss landscape, the error surface, or the objective function surface) is the geometric representation of a loss function as a function of the model's parameters.

Deep LearningMachine Learning

MXFP4

MXFP4 (Microscaling FP4) is a 4-bit block-scaled number format for quantization in machine learning, defined in the Open Compute Project (OCP) Microscaling Formats (MX) Specification v1.0 of September 2023.

AI HardwareAI Inference

Mini-batch stochastic gradient descent

Mini-batch stochastic gradient descent (often shortened to mini-batch SGD or MB-SGD) is the optimization algorithm used to train almost every modern neural network: it updates a model's parameters by…

Minitron

Minitron is a family of compact language models from NVIDIA, together with the model-compression method used to build them: take one large, already pretrained large language model and shrink it into smaller…

NVIDIASmall Language Models

Mistral Forge

Mistral Forge, branded by its developer simply as Forge, is an enterprise platform and engineering service from Mistral AI for building custom large language models and related multimodal models on an…

AI Tools & ProductsEnterprise AI

Model Predictive Control

Model Predictive Control (MPC), also called receding horizon control (RHC), is a feedback control method that, at every sampling instant, solves a finite horizon optimal control problem using the current…

Robotics

Momentum

Momentum is an optimization technique that accelerates gradient descent by accumulating an exponentially weighted moving average of past gradients into a velocity vector and using that velocity, rather than…

Muon (optimizer)

Muon (short for MomentUm Orthogonalized by Newton-Schulz) is a neural-network optimizer that updates the two-dimensional weight matrices of hidden layers by taking the momentum-based SGD update and…

Deep Learning

NaN Trap

A NaN trap (short for "Not a Number" trap) is a failure mode in machine learning training where arithmetic operations produce the special IEEE 754 value NaN, which then propagates through all subsequent…

Deep LearningMachine Learning

NormalFloat 4-bit (NF4)

NormalFloat 4-bit (NF4) is a 4-bit numerical data type for storing the weights of deep neural networks, introduced in the 2023 QLoRA paper by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer…

AI Inference

ORPO

ORPO (Odds Ratio Preference Optimization) is a preference alignment algorithm for large language models that merges supervised fine-tuning and preference alignment into a single training stage, eliminating the…

Large Language ModelsMachine Learning

Objective

In machine learning, an objective (or objective function) is the scalar function that a learning algorithm optimizes during training: the single number whose gradients drive every parameter update.

Machine Learning

Objective function

An objective function is the single scalar-valued quantity that an optimization algorithm tries to minimize or maximize during training.

Olmo-core

Olmo-core is an open-source Python library for modeling and large-scale distributed training with PyTorch. The Allen Institute for AI (Ai2) develops and maintains it under the Apache 2.0 license.

AI InfrastructureDeveloper Tools

Optimization

Optimization (also called mathematical optimization) is the study of choosing the best value of a set of variables according to a numerical criterion, usually by minimizing or maximizing an objective function…

Machine LearningMathematics

Optimizer

An optimizer in machine learning is an algorithm that iteratively adjusts a model's learnable parameters to minimize (or maximize) an objective function, commonly called a loss function

Deep LearningMachine Learning

PEFT

Parameter-efficient fine-tuning (PEFT) is a family of methods that adapt a pre-trained large language model to a downstream task by updating only a small fraction of its parameters

Parameter Server (PS)

The Parameter Server (PS) is a distributed system architecture for training large machine learning models in which one set of machines, the server nodes, holds the global model parameters as a partitioned…

MLOps

Parameter update

A parameter update is the step in neural-network training where a model's trainable weights are adjusted using the gradient of the loss.

Partitioning strategy

A partitioning strategy in distributed deep learning is the plan that decides how a model and its training data are split across multiple accelerators (typically GPUs or TPUs) so that the workload can be…

MLOps

PiSSA

PiSSA (Principal Singular values and Singular vectors Adaptation) is a parameter-efficient fine-tuning method for large language models that initializes LoRA-style low-rank adapter matrices from the dominant…

Large Language Models

Pre-Trained Model

A pre-trained model is a machine learning model that has already been trained on a large, general-purpose dataset and can then be reused, either as a fixed feature extractor or by fine-tuning

Deep LearningMachine Learning

Pruning

Pruning is a family of techniques used in machine learning and artificial intelligence to remove parts of a model or search space that are estimated to be unnecessary for accuracy or optimality.

AI InferenceMachine Learning

QLoRA

QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…

AI InferenceDeep Learning

RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) is a post-training paradigm for large language models in which the reward signal comes from a deterministic

AI InferenceReasoning Models

RMSProp

RMSProp (Root Mean Square Propagation) is an adaptive learning-rate optimizer that divides each parameter's gradient by a running root-mean-square of that parameter's recent gradients

Deep Learning