Reinforcement Learning

Process reward model (PRM)

A process reward model (PRM), also called a process-supervised reward model or step-level verifier, is a learned scoring model that evaluates the correctness...

AI SafetyMachine Learning

Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO) is an on-policy policy gradient reinforcement learning algorithm that stabilizes training by clipping the policy update so...

Machine LearningTraining & Optimization

Q-Function

The Q-function, also called the action-value function or state-action value function and written , is the function in reinforcement learning (RL) that returns...

Machine Learning

Q-Learning

See also: Machine learning terms Q-learning is a model-free, off-policy reinforcement learning algorithm that learns the value of taking a given action in a...

Machine Learning

Quiet-STaR

Quiet-STaR is a self-supervised training method that teaches a large language model to generate short, token-level internal "thoughts," or rationales, that...

Machine Learning

RLAIF

Reinforcement Learning from AI Feedback (RLAIF) is a family of alignment techniques for large language models in which the preference labels used to fine-tune...

AI SafetyMachine Learning

RLOO (REINFORCE Leave-One-Out)

RLOO (REINFORCE Leave-One-Out) is an online reinforcement learning algorithm for aligning large language models with reward signals such as those derived from...

AI AlignmentTraining & Optimization

RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) is a post-training paradigm for large language models in which the reward signal comes from a...

AI InferenceReasoning Models

Random Policy

See also: Reinforcement Learning, Policy, Epsilon Greedy Policy, Q-Learning A random policy is a reinforcement learning policy that chooses actions from a...

Machine Learning

ReST / ReST-EM (Reinforced Self-Training)

ReST (Reinforced Self-Training) is a family of self-training algorithms for large language models that improve a model by fine-tuning it on its own filtered...

Machine Learning

Recursive reward modeling

Recursive reward modeling (RRM) is a proposed approach to the scalable oversight problem in AI alignment, in which agents trained by reward modeling are...

AI AlignmentAI Safety

Reinforcement learning

Reinforcement learning (RL) is a branch of machine learning in which an agent learns to make decisions by taking actions in an environment to maximize a...

Artificial IntelligenceDeep Learning

Replay Buffer

A replay buffer (also called an experience replay buffer or replay memory) is a fixed-size memory that stores an off-policy reinforcement learning agent's past...

Deep LearningMachine Learning

Return (Reinforcement Learning)

In reinforcement learning, the return (commonly denoted ) is the total cumulative reward an agent receives from time step onward, usually with future...

Machine Learning

Reward

See also: Reinforcement Learning, Policy, Q-Learning, Bellman Equation In reinforcement learning (RL), a reward is a scalar feedback signal that an environment...

Machine Learning

Reward Model

A reward model (RM) is a model trained to score the outputs of another AI system, producing a scalar estimate of how good a candidate response is according to...

Reward hacking

Reward hacking (also called specification gaming) is a failure mode in artificial intelligence in which a system maximizes its given objective or reward signal...

AI AlignmentAI Safety

RewardBench

RewardBench is a benchmark and public leaderboard for evaluating reward models, the scoring functions that sit at the center of reinforcement learning from...

AI BenchmarksModel Evaluation

Richard S. Sutton

Richard Stuart Sutton (born 1957 or 1958) is a Canadian-American computer scientist regarded as one of the founders of modern computational reinforcement...

AI ResearchPeople

Robot learning

Robot learning is a field at the intersection of robotics and machine learning in which robots acquire new skills, adapt to new environments, and improve their...

Deep LearningMachine Learning

Robotics Models

Robotics models are machine learning systems that give robots the ability to perceive their surroundings, plan actions, and execute motor control. They span...

AI Models

SARSA (State-Action-Reward-State-Action)

SARSA (State-Action-Reward-State-Action) is an on-policy temporal-difference (TD) control algorithm that learns the action-value function Q^pi(s, a) of the...

Machine Learning

STaR (Self-Taught Reasoner)

STaR (Self-Taught Reasoner) is a self-training method that teaches a large language model to reason by having it generate its own chain-of-thought rationales,...

Machine Learning

Selective Language Modeling (Rho-1)

Selective Language Modeling (SLM) is a pretraining objective for language models that applies the training loss to only a chosen subset of tokens rather than...

Machine Learning

Sergey Levine

Sergey Levine is an American computer scientist, associate professor of electrical engineering and computer sciences at the University of California, Berkeley,...

PeopleRobotics

Sim-to-real transfer

Sim-to-real transfer (also written as sim2real) is the process of training a policy, controller, or perception model inside a physics simulator and deploying...

Robotics

Simulation (in AI and robotics)

Simulation in artificial intelligence and robotics is the use of computational physics, rendering, and procedural environments to recreate a synthetic version...

Robotics

Soft Actor-Critic

Soft Actor-Critic (SAC) is an off-policy, maximum-entropy deep reinforcement learning algorithm that trains a stochastic actor-critic to maximize expected...

AlgorithmsDeep Learning

Sparse upcycling

Sparse upcycling is a technique for building a sparsely activated mixture of experts (MoE) model by initializing it from an already trained dense Transformer...

Machine Learning

Specification gaming

Specification gaming is the phenomenon in which an optimizer satisfies the literal specification of an objective without producing the outcome that the...

AI AlignmentAI Safety

Spinning Up

Spinning Up in Deep RL is a free, open-source educational resource produced by OpenAI to make deep reinforcement learning (deep RL) easier to learn. Released...

OpenAI

State (Reinforcement Learning)

In reinforcement learning (RL), a state is a complete description of the environment at a particular point in time, containing all the information an agent...

Machine Learning

State-Action Value Function

The state-action value function, written Q^π(s, a) and also called the action-value function or Q-function, gives the expected discounted return an agent...

Machine Learning

TIES-Merging

TIES-Merging is a training-free model merging method that combines several models fine-tuned from a shared pre-trained checkpoint into one multitask model...

Machine Learning

Tabular Q-Learning

Tabular Q-learning is the classic form of Q-learning, a model-free reinforcement learning algorithm that stores the action-value function Q(s, a) explicitly in...

Machine Learning

Target Network

A target network is a separate, slowly updated copy of a neural network used in deep reinforcement learning to compute stable learning targets, decoupling the...

Deep LearningMachine Learning

Task arithmetic

Task arithmetic is a model-editing technique that steers the behavior of a neural network by adding or subtracting vectors in its weight space. The central...

Machine Learning

Temporal-difference learning

Temporal-difference (TD) learning is a class of model-free reinforcement learning methods that learn value-function estimates by bootstrapping: updating each...

AlgorithmsMachine Learning

Tim Rocktäschel

Tim Rocktäschel is a German computer scientist known for his work on reinforcement learning, open-ended learning, and language-based AI agents. He is a...

AI AgentsPeople

Trajectory (Reinforcement Learning)

A trajectory in reinforcement learning is a sequence of states, actions, and rewards that an agent experiences while interacting with an environment. Formally...

Machine Learning

Twin Delayed DDPG

Twin Delayed Deep Deterministic Policy Gradient (TD3) is an off-policy actor-critic reinforcement learning algorithm for continuous action spaces, introduced...

AlgorithmsDeep Learning

Tülu 3

Tülu 3 is a fully open post-training recipe and a corresponding family of instruction-tuned language models released by the Allen Institute for AI (Ai2) on...

AI ResearchOpen Source AI

VAPO (Value-based Augmented PPO)

VAPO (Value-based Augmented Proximal Policy Optimization) is a reinforcement learning framework for training large language models on long chain-of-thought...

Machine Learning

π*0.6 (pi-star-0.6)

π0.6 (written "Pi-star-0.6") is a vision-language-action robot foundation model developed by Physical Intelligence, a San Francisco robotics startup. Announced...

AI ModelsRobotics