Reinforcement learning

Reinforcement learning (RL) is a branch of machine learning in which an agent learns to make decisions by taking actions in an environment to maximize a cumulative reward signal, discovering good behavior through trial and error rather than from labeled examples.^[1] Richard Sutton and Andrew Barto, whose textbook is the standard reference in the field,^[51] define it as learning what to do, that is, "how to map situations to actions," so as "to maximize a numerical reward signal."^[1] They treat it as a third machine learning paradigm, alongside supervised learning and unsupervised learning.^[1] Sutton and Barto received the 2024 Turing Award, often called the Nobel Prize of computing, for developing the conceptual and algorithmic foundations of reinforcement learning.^[7]

Unlike supervised learning, which requires labeled input/output pairs, and unlike unsupervised learning, which focuses on finding hidden structure in unlabeled data, reinforcement learning focuses on finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge) through trial-and-error interaction with an environment.^[3] The environment is typically formulated as a Markov decision process (MDP), as many reinforcement learning algorithms utilize dynamic programming techniques.^[4] RL has driven some of the most visible achievements in modern artificial intelligence, from defeating world champions at board games and video games to aligning large language models with human preferences.

Overview

Reinforcement learning achieved widespread recognition through several landmark achievements. In 2016, DeepMind's AlphaGo defeated world champion Lee Sedol in the complex game of Go, a result DeepMind describes as coming "a decade before experts thought possible."^[58] In 2019, OpenAI Five defeated the reigning world champion team in Dota 2^[6], demonstrating RL's ability to handle complex team-based strategy games. By 2022-2023, RL had become central to the training of large language models like ChatGPT and Claude through a technique called Reinforcement Learning from Human Feedback (RLHF) and related methods.^[32][77]

The field emerged from the convergence of multiple intellectual traditions. The psychology of animal learning, beginning with Edward Thorndike's Law of Effect in 1911, established that behaviors followed by satisfying consequences tend to be repeated. The mathematical framework came from optimal control theory and Richard Bellman's development of dynamic programming in the 1950s. These threads were unified in the modern field through the work of Richard Sutton and Andrew Barto, who received the 2024 Turing Award for their foundational contributions.^[7] The Turing Award, presented by the Association for Computing Machinery (ACM), carries a 1 million US dollar prize funded by Google. In its announcement, the ACM said that in a series of papers beginning in the 1980s, Barto and Sutton "introduced the main ideas, constructed the mathematical foundations, and developed important algorithms for reinforcement learning."^[51] Their textbook, Reinforcement Learning: An Introduction, has been cited more than 75,000 times.^[51]

History

The history of reinforcement learning spans over a century, drawing from psychology, control theory, and computer science. Three distinct intellectual threads developed independently before merging into the unified field recognized today.

Behavioral psychology and trial-and-error learning

The earliest roots of reinforcement learning lie in experimental psychology. Ivan Pavlov's work on classical conditioning in the 1890s and 1900s demonstrated that animals could learn to associate stimuli with rewards, forming the basis for understanding learned behavior.^[8] Edward Thorndike formalized this in 1911 with his Law of Effect, which states that responses followed by satisfying outcomes become more firmly associated with the situation, while responses followed by discomfort become less likely.^[9] B.F. Skinner extended these ideas in the 1930s through operant conditioning, which studied how rewards and punishments shape voluntary behavior. These psychological principles directly inspired the reward-based learning framework that RL uses today.

Optimal control and dynamic programming

The second thread came from applied mathematics. In the 1950s, Richard Bellman developed dynamic programming as a method for solving multi-stage decision problems. His key insight, formalized in the Bellman equation (1957), was that an optimal policy can be decomposed into an immediate decision plus the optimal policy from the resulting state onward.^[10] This recursive formulation became the mathematical backbone of most RL algorithms. According to Sutton and Barto, the terms "reinforcement" and "reinforcement learning" first appeared in the engineering literature in the 1960s (for example, Waltz and Fu, 1965, and Mendel, 1966). Marvin Minsky's 1961 paper "Steps Toward Artificial Intelligence" was particularly influential: it discussed what Minsky called the basic credit-assignment problem, how to distribute credit for success among the many decisions that may have produced it.^[1]

Temporal difference learning and unification

The third thread, temporal difference learning (TD learning), bridged the gap between the other two. TD ideas appeared earlier, in Arthur Samuel's 1959 checkers program and in Ian Witten's work in the 1970s, but in 1988 Richard Sutton separated TD learning from control, treated it as a general prediction method, and introduced the TD(lambda) algorithm.^[1][11] TD methods are model-free and learn by bootstrapping from current value estimates rather than waiting for final outcomes; they combine the sampling approach of Monte Carlo methods with the bootstrapping of dynamic programming, giving practical algorithms for environments where the full model is unknown. Sutton and Barto write that the temporal-difference and optimal-control threads were fully brought together in 1989 with Christopher Watkins's Q-learning.^[1] Their 1998 textbook, Reinforcement Learning: An Introduction, synthesized these threads into a coherent framework and became the standard reference for the field.^[12][51]

Timeline of major developments

What is reinforcement learning, in plain terms?

Reinforcement learning teaches an agent through consequences. The agent is never told the correct action; instead it tries actions, observes the resulting reward, and gradually shifts its behavior toward whatever earns the most reward over time. Sutton and Barto call "trial-and-error search and delayed reward" the two most important distinguishing features of reinforcement learning.^[1] The learner must discover which actions are valuable by trying them, and because a single action can affect not only the immediate reward but also the next situation and therefore all subsequent rewards, the agent must reason about long-term consequences rather than instantaneous payoff.^[1]

Core concepts

How does reinforcement learning work? Agent-environment interaction

Reinforcement learning problems involve an agent interacting with an environment through a cycle of observation, action, and reward.^[3] At each discrete time step t:

1. The agent observes the current state s_t of the environment.
2. Based on its policy pi, the agent selects an action a_t.
3. The environment transitions to a new state s_{t+1} according to transition probabilities P(s' | s, a).
4. The agent receives a scalar reward r_{t+1} indicating the immediate benefit of that action.

The agent's objective is to learn a policy that maximizes the expected return (cumulative discounted reward):^[1]

G_t = R_{t+1} + gamma * R_{t+2} + gamma^2 * R_{t+3} + ... = Sum_{k=0}^{infinity} gamma^k * R_{t+k+1}

The discount factor gamma (where 0 <= gamma <= 1) controls how much the agent values future rewards relative to immediate ones. A gamma close to 0 makes the agent short-sighted, prioritizing immediate reward. A gamma close to 1 makes the agent far-sighted, weighting future rewards almost as heavily as immediate ones. Choosing the right discount factor is problem-dependent: a robot navigating a maze might use gamma = 0.99, while a day-trading algorithm might use a lower value.

Key components

Value functions

Value functions are central to reinforcement learning, estimating how good it is for an agent to be in a particular state or to take a particular action in a state:^[1]

• State-value function V^pi(s): the expected return starting from state s and following policy pi.
• Action-value function Q^pi(s,a): the expected return from taking action a in state s, then following policy pi.

The optimal value functions satisfy the Bellman optimality equations:^[4]

• V(s) = max_a Sum_{s'} P(s'|s,a) [R(s,a,s') + gamma * V*(s')]*
• Q(s,a) = Sum_{s'} P(s'|s,a) [R(s,a,s') + gamma * max_{a'} Q*(s',a')]*

These equations express the key recursive insight: under an optimal policy, the value of a state equals the expected return of the best action available in it, that is, the expected immediate reward plus the discounted value of the resulting next state.

Exploration vs. exploitation

One fundamental challenge in reinforcement learning is the exploration-exploitation tradeoff.^[2] The agent must balance:

• Exploration: trying new, untested actions to discover potentially better strategies.
• Exploitation: using what the agent already knows to maximize immediate rewards.

An agent that only exploits may get stuck in a suboptimal policy, never discovering better options. An agent that only explores wastes time on actions it already knows are bad. Common strategies for managing this tradeoff include:

What is the difference between reinforcement learning and supervised learning?

Reinforcement learning and supervised learning are both branches of machine learning, but they differ in where the training signal comes from and what the model is asked to do. Supervised learning is trained on a fixed dataset of correct input/output pairs and learns to imitate those labels; reinforcement learning has no correct answers, only a scalar reward, and must generate its own data by acting in an environment.^[1][2] In supervised learning each example is independent, whereas in RL an action changes the agent's future situation, so decisions are sequential and rewards can be delayed. The table below summarizes the main differences.

In practice the two paradigms are often combined: many systems are first trained with supervised learning and then refined with reinforcement learning. The training of ChatGPT is a prominent example, beginning with supervised fine-tuning on human demonstrations and then applying RL to optimize a reward model that captures human preferences.^[31]

Mathematical foundations

Markov decision processes

Reinforcement learning problems are formally modeled as Markov decision processes (MDPs), defined by the tuple (S, A, P, R, gamma):^[13]

• S: set of states (state space)
• A: set of actions (action space)
• P(s'|s,a): state transition probability function
• R(s,a,s'): reward function
• gamma: discount factor (0 <= gamma < 1)

The Markov property states that the future depends only on the current state, not on the sequence of events that preceded it: P(s_{t+1} | s_t, a_t, s_{t-1}, ..., s_0) = P(s_{t+1} | s_t, a_t). This memoryless property is what makes MDPs tractable. In practice, many real-world problems violate the Markov property (the current observation does not fully capture the state), leading to partially observable MDPs (POMDPs), which are substantially harder to solve.

Bellman equations

The Bellman equations, named after Richard Bellman, provide the recursive decomposition that underpins nearly all RL algorithms. For a given policy pi:

V^pi(s) = Sum_a pi(a|s) Sum_{s'} P(s'|s,a) [R(s,a,s') + gamma * V^pi(s')]

This equation says that the value of a state under policy pi equals the expected immediate reward plus the discounted value of the next state, averaged over all possible actions and transitions. The Bellman optimality equation replaces the policy average with a maximum, defining what the best possible policy would achieve.

Temporal difference learning

Temporal difference (TD) learning, introduced by Sutton in 1988, is a core method in RL that combines ideas from Monte Carlo methods and dynamic programming.^[11] Instead of waiting until the end of an episode to update value estimates (as Monte Carlo methods do), TD methods update estimates after each step using the observed reward and the current estimate of the next state's value:

V(s_t) <- V(s_t) + alpha [r_{t+1} + gamma * V(s_{t+1}) - V(s_t)]

The term in brackets, r_{t+1} + gamma * V(s_{t+1}) - V(s_t), is called the TD error. It measures the difference between the estimated value and a better estimate derived from the actual reward received plus the next state's estimated value. TD learning converges to the true value function under certain conditions and forms the basis of algorithms like Q-learning and SARSA.

Algorithm taxonomy

RL algorithms can be classified along several axes. Understanding these distinctions is essential for choosing the right algorithm for a given problem.

Model-based vs. model-free

Model-free algorithms learn a policy or value function directly from experience without building an explicit model of how the environment works. Q-learning and PPO are model-free. They are simpler to implement but often require many more interactions with the environment.

Model-based algorithms learn or are given a model of the environment's dynamics (transition probabilities and rewards) and use it for planning. Dyna-Q, introduced by Sutton in 1990, was an early approach that combined real experience with simulated experience generated from a learned model.^[14] More recent model-based methods include MuZero, which learns a latent dynamics model focused on predicting rewards and values rather than raw observations,^[66] and Dreamer, which learns a world model in latent space and uses it to train a policy entirely through imagined rollouts.^[15]

Model-based methods tend to be more sample-efficient because they can generate synthetic training data through mental simulation. However, if the learned model is inaccurate, compounding errors can lead to poor policies.

Value-based vs. policy-based

Value-based methods (Q-learning, DQN) learn a value function and derive a policy from it (e.g., always choose the action with the highest Q-value). They work well for discrete action spaces but struggle with continuous actions.

Policy-based methods (REINFORCE, PPO) directly parameterize and optimize the policy without necessarily learning a value function. They handle continuous action spaces naturally and can learn stochastic policies, but tend to have higher variance in gradient estimates.

Actor-critic methods combine both: an actor (policy network) selects actions while a critic (value network) evaluates them. This reduces variance compared to pure policy gradient methods while retaining the ability to handle continuous actions.

On-policy vs. off-policy

On-policy algorithms (SARSA, PPO, A2C) learn about the policy currently being executed. They use data generated by the current policy to update that same policy. This can be more stable but is less sample-efficient because old data cannot be reused after a policy update.

Off-policy algorithms (Q-learning, DQN, SAC) can learn from data generated by any policy, including old versions of the agent or even random exploration. This allows experience replay, where past transitions are stored in a buffer and sampled repeatedly, greatly improving sample efficiency.

Key algorithms

Q-learning

Q-learning, introduced by Christopher Watkins in his 1989 Cambridge PhD thesis Learning from Delayed Rewards, is a model-free, off-policy algorithm that learns the optimal action-value function directly.^[16] The update rule is:

Q(s,a) <- Q(s,a) + alpha [r + gamma * max_{a'} Q(s',a') - Q(s,a)]

where alpha is the learning rate. The key insight is that the update uses the maximum Q-value over the next state's actions regardless of which action the agent actually took. This "off-policy" property means Q-learning can learn about the optimal policy while following an exploratory one. Watkins and Dayan (1992) made rigorous the convergence proof outlined in Watkins's thesis, showing that tabular Q-learning converges to the optimal Q-function with probability 1, given sufficient exploration and decreasing learning rates.^[1][16]

Q-learning is simple and effective for problems with small, discrete state and action spaces. For larger problems, function approximation (such as neural networks) is needed.

SARSA

SARSA (State-Action-Reward-State-Action), introduced by Rummery and Niranjan in 1994, is an on-policy variant of Q-learning.^[17] They called it "Modified Connectionist Q-learning"; the name Sarsa was introduced by Sutton in 1996.^[1] Its update rule uses the action actually taken in the next state rather than the maximum:

Q(s,a) <- Q(s,a) + alpha [r + gamma * Q(s',a') - Q(s,a)]

Because SARSA evaluates the policy it is actually following, it tends to learn safer policies than Q-learning. In a cliff-walking problem, for example, Q-learning learns the optimal path along the cliff edge, while SARSA learns a safer path further from the edge, because it accounts for the possibility of exploratory actions leading to a fall.

Deep Q-Networks (DQN)

Deep Q-Networks (DQN), published by Mnih et al. at DeepMind in 2013 and in Nature in 2015, used deep convolutional neural networks to approximate Q-values for high-dimensional state spaces.^[18][60] DQN took raw pixel inputs from Atari 2600 games and learned to play 49 different games using the same architecture and hyperparameters, scoring more than 75% of a professional human tester's score (the paper's threshold for human-level play) on 29 of them.^[18] The Nature paper reported that "the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters."^[18]

Two innovations made this possible:

• Experience replay: the agent stores transitions (s, a, r, s') in a replay buffer and samples random mini-batches for training. This breaks correlations between consecutive samples and improves data efficiency.
• Target network: a separate, periodically updated copy of the Q-network computes target values. This stabilizes training by preventing the target from shifting with every update.

The 2013 paper described DQN as "the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning."^[60]

Policy gradient methods

Policy gradient methods directly optimize a parameterized policy by estimating the gradient of expected return with respect to policy parameters.^[19] The foundational algorithm is REINFORCE (Williams, 1992), which updates policy parameters theta using:

nabla_theta J(theta) ~ Sum_t G_t * nabla_theta log pi_theta(a_t | s_t)

where G_t is the return from time step t. The intuition is straightforward: increase the probability of actions that led to high returns, decrease the probability of actions that led to low returns.

REINFORCE is simple but suffers from high variance in gradient estimates. Adding a baseline (typically the state value function) reduces variance without introducing bias:

nabla_theta J(theta) ~ Sum_t (G_t - V(s_t)) * nabla_theta log pi_theta(a_t | s_t)

The term (G_t - V(s_t)) is called the advantage, and this leads to the family of advantage actor-critic methods.

Actor-critic methods

Actor-critic algorithms combine policy-based and value-based learning:^[20]

• The actor is a policy network that selects actions.
• The critic is a value network that estimates the value of states or state-action pairs.

The critic reduces variance in the policy gradient estimate by providing a learned baseline. Several important variants exist:

A2C (Advantage Actor-Critic) uses the advantage function A(s,a) = Q(s,a) - V(s) to update the actor. A3C (Asynchronous Advantage Actor-Critic), introduced by Mnih et al. in 2016, runs multiple agents in parallel on separate copies of the environment, each contributing gradients asynchronously to a shared model.^[21] On Atari it surpassed the previous state of the art while training for half the time on a single multi-core CPU instead of a GPU.^[21]

DDPG (Deep Deterministic Policy Gradient), introduced by Lillicrap et al. in 2015, extends DQN to continuous action spaces by learning a deterministic policy alongside a Q-function.^[22] It uses experience replay and target networks, similar to DQN.

TD3 (Twin Delayed DDPG), published by Fujimoto et al. in 2018, addresses overestimation bias in DDPG by maintaining two critic networks and taking the minimum of their estimates, delaying policy updates, and adding noise to target actions.^[23]

Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO), introduced by Schulman et al. at OpenAI in 2017, constrains policy updates to prevent destructively large changes.^[24] PPO optimizes a clipped surrogate objective:

L^CLIP(theta) = E[min(r_t(theta) * A_t, clip(r_t(theta), 1 - epsilon, 1 + epsilon) * A_t)]

where r_t(theta) = pi_theta(a_t | s_t) / pi_{theta_old}(a_t | s_t) is the probability ratio and epsilon is typically 0.2. The clipping prevents the new policy from deviating too far from the old one in a single update.

PPO has become one of the most widely used RL algorithms due to its simplicity, stability, and strong empirical performance; shortly after its release OpenAI called it "the default reinforcement learning algorithm at OpenAI."^[64] OpenAI used it to train OpenAI Five (Dota 2) and for the RLHF stage of InstructGPT and ChatGPT.^[62][31][77]

Soft Actor-Critic (SAC)

Soft Actor-Critic (SAC), introduced by Haarnoja et al. in 2018, augments the standard RL objective with an entropy term that encourages exploration:^[25]

J(pi) = Sum_t E[r(s_t, a_t) + alpha * H(pi(.|s_t))]

where H is the entropy of the policy and alpha is a temperature parameter controlling the tradeoff between reward maximization and entropy (exploration). SAC is off-policy and uses experience replay; a follow-up version (Haarnoja et al., December 2018) added automatic tuning of the temperature parameter.^[65] It achieves strong performance on continuous control benchmarks with better sample efficiency than on-policy methods like PPO.

Algorithm comparison

Deep reinforcement learning

Deep reinforcement learning (deep RL) combines RL algorithms with deep neural networks as function approximators, enabling agents to handle high-dimensional state and action spaces that are intractable for tabular methods.

Why deep learning transformed RL

Classic RL algorithms like tabular Q-learning maintain a table of values for every state-action pair. This works for problems with small state spaces (a few hundred or thousand states) but fails completely when states are described by images, continuous variables, or other high-dimensional inputs. A single Atari game frame has 210 x 160 pixels with 128 possible colors per pixel, making the raw state space astronomically large.

Neural networks solve this by learning compact, generalizable representations of value functions or policies. A convolutional neural network can process raw pixels and output Q-values or action probabilities, automatically learning relevant features like object positions, velocities, and spatial relationships.

Key architectural patterns

Deep RL uses several recurring architectural patterns:

• Convolutional networks for processing visual observations (DQN, AlphaGo).
• Recurrent networks (LSTMs, GRUs) for handling partial observability and sequential dependencies.
• Transformers for sequence modeling and attention over long histories (Decision Transformer, Gato).
• Residual networks and normalization layers for training stability in deep value and policy networks.

Stability challenges

Combining neural networks with RL introduces several instability issues that do not arise in supervised learning. The training data distribution changes as the policy improves (non-stationarity). Small changes in the value function can cause large changes in the policy, which in turn changes the data distribution. Experience replay, target networks, gradient clipping, and entropy regularization are common techniques for addressing these issues.

Major milestones

TD-Gammon (1992)

TD-Gammon, developed by Gerald Tesauro at IBM's Thomas J. Watson Research Center, was one of the earliest demonstrations that RL combined with neural networks could achieve expert-level performance.^[26] As Sutton and Barto describe it, the program evaluated backgammon positions with a neural network that had 198 input units encoding the board, one hidden layer (40 or 80 units, depending on the version) and an output estimating the probability of winning.^[1] It learned through self-play using TD(lambda); version 2.1 trained on about 1.5 million games against itself.^[1] After a 40-game test against version 2.1, former world champion Bill Robertie judged that it played "at a strong master level that is extremely close (within a few hundredths of a point) to equaling the world's best human players."^[26] The program is commonly cited as a precursor to the deep RL breakthroughs that followed two decades later.

DQN and Atari (2013, 2015)

DeepMind's DQN learned control policies directly from raw pixel inputs.^[18] The 2013 paper applied it to seven Atari games, where it outperformed all previous approaches on six and surpassed a human expert on three;^[60] the 2015 Nature paper extended this to 49 games, reaching more than 75% of a professional human tester's score on 29 of them with an identical architecture and hyperparameters for every game.^[18] A different network was trained on each game, so the result showed that one general method could learn very different tasks, not that a single trained agent could play all of them.^[18]

AlphaGo and AlphaZero (2016, 2017)

AlphaGo defeated Lee Sedol, winner of 18 world Go titles, 4-1 in Seoul in March 2016; DeepMind says the match was watched by over 200 million people.^[58] AlphaGo combined supervised learning from human expert games with RL through self-play, using Monte Carlo tree search (MCTS) guided by a policy network and a value network.^[5]

AlphaGo Zero, published in Nature in October 2017, eliminated the need for human data entirely, learning exclusively through self-play starting from random play.^[27] It outperformed AlphaGo Lee, the version that beat Lee Sedol, after 36 hours of training and beat it 100 games to 0 after 72 hours; a larger version trained for about 40 days then beat AlphaGo Master 89 games to 11.^[27] AlphaZero generalized the approach to chess and shogi. Its December 2017 preprint reported superhuman play in all three games "within 24 hours."^[59] The peer-reviewed Science version (2018) reported that AlphaZero first outperformed Stockfish after about 4 hours of training, Elmo after 2 hours and AlphaGo Lee after 30 hours, and that it "convincingly defeated a world champion program" in each game.^[28]

OpenAI Five (2019)

OpenAI Five tackled Dota 2, which adds challenges absent from Go: imperfect information, real-time decision-making, long time horizons (about 20,000 decision steps per game, with an action every fourth frame), a massive action space, and five-player teamwork.^[6][62] The system was trained with PPO and self-play in a single run from June 2018 to April 2019; at its peak the run used 1,536 GPUs for optimization and up to 172,800 CPUs for game rollouts (a June 2018 version had used 256 GPUs and 128,000 CPU cores).^[62][63] By the finals it had experienced about 45,000 years of Dota 2 self-play over 10 real-time months.^[6] On April 13, 2019, it beat OG, the reigning world champions, 2-0 in a best-of-three match.^[62] OpenAI Five demonstrated that PPO and massive-scale self-play could handle multi-agent coordination in complex real-time environments.

AlphaStar (2019)

DeepMind's AlphaStar reached Grandmaster level in StarCraft II, placing in the top 0.2% of human players on the official European ladder.^[29] StarCraft II presents challenges beyond Go: imperfect information (fog of war), real-time actions, long-term strategic planning, and a combinatorial action space. AlphaStar combined imitation learning from human replays with multi-agent reinforcement learning, training a league of agents that competed against one another to develop diverse strategies.

AlphaProof (2024)

DeepMind's AlphaProof applied reinforcement learning to formal mathematics, pairing an AlphaZero-inspired agent with the Lean proof assistant so that every generated proof is machine-checked and therefore guaranteed correct.^[57] Trained on a curriculum of millions of auto-formalized problems and using test-time RL that generates variants of each target problem, AlphaProof solved three of the five non-geometry problems at the 2024 International Mathematical Olympiad, including the competition's hardest problem. Combined with AlphaGeometry 2, and with multi-day computation, the system reached a score equivalent to a silver medallist, which the authors describe as the first time an AI system achieved any medal-level performance, "to our knowledge."^[57] The work was published online in Nature in November 2025 and extended the AlphaZero self-play recipe from games with fixed rules to the open-ended domain of theorem proving.^[57]

Reinforcement Learning from Human Feedback (RLHF)

RLHF has become one of the most consequential applications of reinforcement learning. It is the technique that transforms a pre-trained language model into a conversational assistant that follows instructions, refuses harmful requests, and generally behaves in ways humans find helpful.^[30]

How RLHF works

The RLHF process typically involves three stages:

1. Supervised fine-tuning (SFT): a pre-trained language model is fine-tuned on a dataset of human-written demonstrations of desired behavior.
2. Reward model training: human labelers compare pairs of model outputs and indicate which they prefer. These preference labels train a reward model that predicts a scalar score for any given output.
3. RL optimization: the SFT model is further trained using RL (typically PPO) to maximize the reward model's score, with a KL-divergence penalty to prevent the model from deviating too far from the SFT model.

OpenAI's InstructGPT (2022) was one of the first published demonstrations of this approach for instruction following,^[31] and OpenAI said it trained ChatGPT "using the same methods as InstructGPT, but with slight differences in the data collection setup."^[77] Anthropic applied a variant called Constitutional AI (CAI) to train Claude, where AI-generated feedback partially replaces human labeling.^[32]

Evolution beyond PPO

The RL component of RLHF has evolved rapidly:

GRPO was introduced in February 2024 in DeepSeek's DeepSeekMath paper as a memory-efficient variant of PPO for improving mathematical reasoning. It removes the separate value (critic) network, instead normalizing each response's reward against the mean and standard deviation of the rewards for a group of responses sampled for the same prompt.^[52] In DeepSeekMath itself, GRPO optimized against a learned reward model.^[52] This critic-free design later proved well suited to RL on LLMs, where sampling many completions per prompt is cheap and training a stable value function over long token sequences is difficult.

DeepSeek-R1, released in January 2025, demonstrated that RL using GRPO with verifiable rewards can produce strong reasoning capabilities in LLMs.^[33] Its companion model, DeepSeek-R1-Zero, was trained without any supervised fine-tuning step, using rule-based rewards for answer accuracy and output format, and developed behaviors such as self-reflection, verification, and increasingly long chain-of-thought reasoning through RL alone. The released DeepSeek-R1 model itself incorporated cold-start supervised fine-tuning stages before its RL training, and DeepSeek reported performance comparable to OpenAI's o1-1217 on reasoning tasks.^[33]

Reinforcement Learning with Verifiable Rewards (RLVR)

RLVR is a training paradigm where rewards come from deterministic, rule-based verifiers rather than learned reward models.^[54] For mathematical problems, the verifier checks whether the model's final answer matches the correct solution. For code generation, automated tests serve as the verifier. DeepSeek said it avoided neural reward models for R1's reasoning tasks because they "are susceptible to reward hacking during large-scale reinforcement learning,"^[33] and the authors of Dr. GRPO note that RL tuning of reasoning models "typically employs rule-based verifiers."^[67] Rule-based rewards were used in DeepSeek-R1 (optimized with GRPO) and in ByteDance's DAPO (a GRPO variant), among other 2025 recipes for open reasoning models.^[33][55] The term "reinforcement learning with verifiable rewards" was coined in November 2024 by the Allen Institute for AI in its Tulu 3 post-training work, which explains that RLVR "leverages the existing RLHF objective but replaces the reward model with a verification function."^[54] DeepSeekMath, which introduced GRPO earlier that year, had trained against a learned reward model, comparing rule-based answer checking only in its analysis.^[52]

Process and outcome reward models

A key distinction in RL for reasoning is what the reward signal grades. An outcome reward model (ORM) scores only the final answer, while a process reward model (PRM) scores each intermediate step of a chain of thought.^[53] OpenAI's 2023 study "Let's Verify Step by Step" (Lightman et al.) trained a PRM on PRM800K, a dataset of roughly 800,000 human step-level correctness labels on solutions to MATH problems, and found that step-level process supervision produced a more reliable reward model than outcome-only supervision, with the process-supervised verifier solving 78% of a representative MATH test subset.^[53] PRMs give denser feedback and can localize where a derivation went wrong, but they are expensive to label and can be gamed. DeepSeek-R1 deliberately avoided both outcome-based and process-based neural reward models for reasoning tasks, relying on rule-based accuracy and format rewards, and still developed sophisticated multi-step reasoning.^[33]

GRPO and its variants

The rapid adoption of GRPO for reasoning exposed several weaknesses that a family of successor algorithms set out to fix; DAPO and Dr. GRPO, for example, both drop the KL-penalty term from the original objective.^[55][67] DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization), released open-source by ByteDance's Seed team in 2025, decouples the upper and lower clipping bounds and dynamically filters prompts to keep informative samples, reaching 50 points on AIME 2024 with a Qwen2.5-32B base model.^[55] Dr. GRPO (2025), from Sea AI Lab and university collaborators, removes the per-response length normalization and the standard-deviation normalization from GRPO's objective; the authors found that GRPO's bias "artificially increases response length (especially for incorrect outputs) during training."^[67] GSPO (Group Sequence Policy Optimization), introduced by the Qwen team in 2025, computes the importance-sampling ratio and applies clipping at the level of the whole generated sequence rather than per token, which stabilizes training on mixture-of-experts models; the Qwen team says it contributed to the performance gains of the latest Qwen3 models in mid-2025.^[56]

What is reinforcement learning used for?

Reinforcement learning is used wherever a system must learn a sequence of decisions to optimize a long-term outcome rather than predict a single label. Its highest-profile successes are in game playing, but the same machinery now trains industrial control systems, robots, recommendation engines, and the alignment of large language models. The sections below survey the major application areas.

Game playing

Reinforcement learning has achieved superhuman performance in numerous games:

• Board games: AlphaGo (Go) and AlphaZero and MuZero (Go, chess, and shogi) reached superhuman play by combining self-play RL with tree search.^[5][28][66]
• Video games: DQN learned 49 Atari games with a single architecture;^[18] OpenAI Five beat the Dota 2 world champions;^[6] AlphaStar reached Grandmaster in StarCraft II.^[29]
• Poker: Pluribus (2019) proved stronger than top human professionals in six-player no-limit Texas Hold'em; its authors noted that earlier AI successes in poker had been limited to two-player games.^[34]
• Diplomacy: Meta's Cicero (2022) achieved human-level performance in the board game Diplomacy, combining RL with natural language generation for negotiation.^[35]

Robotics

RL enables robots to acquire motor skills through trial and error rather than manual programming:

• Locomotion: policies trained in simulation transfer to physical robots for walking, running, and navigating uneven terrain. Boston Dynamics has integrated RL into the locomotion control system of its Spot robot.^[80]
• Manipulation: OpenAI demonstrated a robotic hand (Dactyl) solving a Rubik's Cube using RL with sim-to-real transfer and domain randomization (2019).^[36]
• Assembly and manufacturing: industrial robots learn assembly sequences, welding paths, and pick-and-place operations.
• Sim-to-real transfer: training in physics simulators (MuJoCo, Isaac Gym) and transferring to physical hardware remains a major research area. Domain randomization, where simulation parameters are varied during training, helps bridge the gap between simulated and real environments.

Autonomous vehicles

Research on autonomous driving has applied RL to several aspects of driving:

• Path planning and trajectory optimization
• Lane changing and merging decisions
• Adaptive cruise control and following distance
• Intersection and traffic light negotiation

RL is typically combined with imitation learning from human drivers rather than used alone. Waymo researchers, for example, reported in 2022 that adding RL with simple rewards to imitation learning on more than 100,000 miles of urban driving data cut failures by over 38% in the most challenging test scenarios, compared with imitation alone.^[73]

Healthcare

RL applications in medicine include:

• Treatment optimization: dynamic treatment regimes, for example sepsis management in ICUs, where the "AI Clinician" learned intravenous fluid and vasopressor dosing policies offline from retrospective intensive-care records.^[37]
• Drug discovery: molecular design and optimization of chemical structures.
• Personalized medicine: adaptive clinical trial designs that allocate patients to treatments based on observed responses.
• Medical imaging: RL-guided strategies for anatomical landmark detection and image acquisition optimization.

Finance

Financial applications include:

• Algorithmic trading: automated strategies that learn to execute trades, manage inventory, and time entries and exits.
• Portfolio management: dynamic asset allocation that adapts to changing market conditions.
• Risk management: credit scoring models and fraud detection systems.
• Market making: RL agents that provide liquidity and manage bid-ask spreads.

Energy and sustainability

• Data center cooling: DeepMind's 2016 system cut the energy Google used for data center cooling by up to 40% in live tests. DeepMind describes it as an ensemble of deep neural networks that predict energy efficiency and recommend actions, not as reinforcement learning; its 2018 autonomous successor likewise chooses actions from neural-network predictions of energy use under safety constraints, with average savings of about 30%.^[38][69]
• Smart grids: RL for load balancing, demand response, and renewable energy integration.
• Wind farms: optimizing turbine yaw angles and blade pitch to maximize energy output.
• Building management: HVAC and lighting optimization in commercial buildings.

Natural language processing and LLM alignment

• RLHF and RLVR: training ChatGPT, Claude, GPT-4, Gemini, Llama, and DeepSeek to follow instructions and align with human values.
• Dialogue systems: optimizing conversational agents for engagement and task completion.
• Machine translation: improving translation quality through reward signals based on BLEU scores or human preferences.
• Text summarization: generating concise, informative summaries optimized by RL-based reward signals.

Recommendation systems

RL is used in recommendation systems where the goal is to maximize long-term user engagement rather than immediate click-through rates. YouTube, for example, has described a production recommender trained with the REINFORCE policy-gradient algorithm, with an off-policy correction for learning from logged user feedback, and reported live experiments showing the value of exploration.^[74] RL-style approaches let recommenders balance exploration (showing new content) with exploitation (recommending proven favorites) and account for the sequential nature of user interactions.

Multi-agent reinforcement learning

Multi-agent reinforcement learning (MARL) extends RL to settings where multiple agents interact within a shared environment.^[39] This introduces challenges absent in single-agent RL: agents must account for the behavior of other learning agents, which makes the environment non-stationary from each agent's perspective.

Types of multi-agent settings

Key approaches

• Independent learners: each agent runs its own RL algorithm, treating other agents as part of the environment. Simple but ignores the non-stationarity caused by other agents learning simultaneously.
• Centralized training, decentralized execution (CTDE): agents share information during training (e.g., a shared critic with global state) but act based only on local observations during deployment. QMIX and MAPPO are popular CTDE algorithms.
• Communication learning: agents learn to communicate through discrete or continuous messages, enabling coordination in partially observable settings.
• Population-based training: a population of agents with different strategies co-evolve, as used in AlphaStar's league training.

Applications of MARL

MARL has been applied to autonomous driving (multiple vehicles negotiating at intersections), robotic swarms (coordinated exploration and task allocation), traffic signal control (city-wide optimization of traffic flow), multiplayer games (Dota 2, StarCraft II), and resource allocation in smart grids and communication networks. A textbook on MARL by Stefano Albrecht, Filippos Christianos and Lukas Schäfer was published by MIT Press in 2024.^[40]

Challenges and limitations

Sample inefficiency

RL algorithms often require enormous amounts of interaction data to learn effective policies:^[41]

• DQN: 50 million training frames per Atari game, "around 38 days of game experience" according to the Nature paper;^[18] Lake and colleagues put the same budget at 200 million frames, approximately 924 hours of game time per game, against about two hours of practice for the professional human tester.^[61]
• OpenAI Five: about 45,000 years of Dota 2 self-play^[6]
• AlphaGo Zero: 4.9 million self-play games over 3 days for its 20-block version; the larger 40-block version trained for 40 days and generated 29 million games.^[27]

This makes direct training on physical systems (robots, real vehicles) impractical for most current algorithms. Solutions include model-based RL (generating synthetic data from learned models), transfer learning (reusing knowledge from related tasks), curriculum learning (gradually increasing task difficulty), and offline RL (learning from fixed datasets without further interaction).

Exploration in large state spaces

Effective exploration becomes extremely difficult in environments with:

• Sparse rewards: where the agent receives no feedback until it reaches a rare goal state. A robot learning to stack blocks might receive a reward only upon successful completion, with no signal during the thousands of intermediate steps.
• Large or continuous state spaces: where the number of possible configurations is astronomical.
• Safety-critical domains: where exploration risks catastrophic failures. A self-driving car cannot explore bad driving strategies.

Approaches to these challenges include intrinsic motivation and curiosity-driven exploration (rewarding the agent for visiting novel states), hierarchical RL (decomposing problems into subgoals), and safe exploration methods with constraints.

Reward specification and reward hacking

Designing reward functions that capture the true objective is notoriously difficult:^[42]

• Reward hacking: agents exploit unintended shortcuts in the reward function. In OpenAI's 2016 CoastRunners example, a boat-racing agent rewarded for points learned to circle in a lagoon, repeatedly knocking over the same respawning targets instead of finishing the race.^[78]
• Reward shaping: manually engineering intermediate rewards to guide learning is error-prone and can introduce biases.
• Specification gaming: agents find unexpected strategies that satisfy the letter of the reward function but not its intent.

In RLHF for LLMs, reward hacking manifests as models producing verbose, sycophantic responses that score highly with the reward model but are not actually more helpful. Mitigation strategies include inverse RL (learning rewards from demonstrations), reward model ensembles, and reward-shaping methods such as Preference as Reward (PAR), proposed in 2025, which uses the latent preferences encoded in the reward model as a bounded RL signal.^[72]

Sim-to-real transfer

Policies trained in simulation often fail when deployed on physical hardware due to the "sim-to-real gap": differences in physics, sensor noise, actuator dynamics, and visual appearance between simulator and reality.^[43] Domain randomization (varying simulation parameters during training), system identification (calibrating simulation to match reality), and progressive domain adaptation help bridge this gap.

Generalization and catastrophic forgetting

RL agents often fail to generalize beyond their training environment. A policy trained in one version of a video game may fail on a slightly different version. When learning multiple tasks sequentially, neural networks suffer from catastrophic forgetting, where learning a new task overwrites the weights needed for previously learned tasks. Meta-learning, domain randomization, and continual learning are active research areas addressing these issues.

Interpretability and safety

Neural network policies are black boxes; it is difficult to understand why an agent takes a particular action. This creates problems for:

• Verification: proving that an RL system will behave safely in all possible situations.
• Debugging: identifying why an agent fails in specific scenarios.
• Regulation: deploying RL in safety-critical domains like healthcare or autonomous driving requires explainable decision-making.
• AI alignment: ensuring that RL agents' learned objectives align with human values and intentions.

Current research directions (2025-2026)

Offline reinforcement learning

Offline RL (also called batch RL) learns from fixed datasets of previously collected transitions without any further environment interaction.^[44] This is valuable in domains where online exploration is expensive or dangerous (healthcare, autonomous driving, industrial control). Key methods include:

• Conservative Q-Learning (CQL): penalizes Q-values for out-of-distribution actions to prevent overestimation.
• Implicit Q-Learning (IQL): avoids querying out-of-distribution actions entirely.
• Decision Transformer: frames RL as a sequence modeling problem, using a transformer to predict actions conditioned on desired returns.

Foundation models meet RL

The intersection of foundation models and RL is one of the most active research areas. Several directions have emerged:

• RL for foundation model training: RLHF, RLVR, and GRPO for aligning and improving LLMs.
• Foundation models for RL: using pre-trained language and vision models to provide representations, world knowledge, or reward signals for RL agents.
• Generalist agents: DeepMind's Gato (2022) and Google's RT-2 (2023) are large pre-trained models that act across several domains (text, images, robotic control), but both are trained with supervised learning rather than RL. Gato was "trained offline in a purely supervised manner," partly on trajectories generated by specialist RL agents,^[45] and RT-2 co-fine-tunes a vision-language model on robot trajectory data and web-scale vision-language tasks.^[70]
• Vision-Language-Action models: RT-1, RT-2, and similar systems use transformer architectures to map visual observations and language instructions directly to robot actions.

World models and model-based deep RL

Learned world models allow agents to plan and imagine future scenarios without interacting with the real environment:

• Dreamer (v1, v2, v3): learns a latent dynamics model and trains a policy entirely through imagined rollouts, achieving competitive performance with far less real-world data.^[15]
• RLVR-World (2025): a framework that uses reinforcement learning with verifiable rewards to directly optimize world models across domains including text games, web navigation, and robot manipulation.^[71]
• Differentiable physics simulators: enable gradient-based optimization through simulated physics for robotics applications.

Hierarchical reinforcement learning

Hierarchical RL decomposes complex, long-horizon tasks into manageable subtasks:

• Options framework: temporal abstraction through "options" (sub-policies that execute over multiple time steps).
• Goal-conditioned policies: high-level policies set subgoals; low-level policies achieve them.
• Feudal networks: hierarchical architectures where a manager sets goals for a worker.

This is particularly relevant for robotics and navigation tasks where planning over hundreds or thousands of steps is needed.

Safe reinforcement learning

Safe RL develops algorithms that satisfy safety constraints during both training and deployment. Constrained MDPs formalize safety requirements as constraints on expected costs. Shielding approaches use formal verification to block unsafe actions. This is a growing area as RL moves into safety-critical applications like autonomous driving and medical treatment optimization.

Recent developments (2026)

The momentum behind RL for reasoning and agentic models continued through 2026. In September 2025, the DeepSeek-R1 work was published in Nature (vol. 645, pp. 633-638) after peer review, formalizing the claim that reasoning can be incentivized through pure RL with GRPO.^[46] In an accompanying editorial, Nature's editors noted that eight specialists had reviewed the work and that none of the most widely used LLMs had previously faced independent peer review in a research journal.^[68] On April 24, 2026, DeepSeek released a preview of its next-generation open-weight model series, DeepSeek-V4, in two MIT-licensed variants: V4-Pro (a 1.6-trillion-parameter mixture-of-experts model with 49 billion activated parameters) and the smaller V4-Flash (284 billion parameters, 13 billion activated), both with a one-million-token context window.^[79] Its model card describes a two-stage post-training recipe that cultivates domain experts through supervised fine-tuning and RL with GRPO before consolidating them via on-policy distillation.^[47]

A distinct research direction, "agentic RL," consolidated during 2025-2026. A survey first posted in September 2025 and revised through 2026 contrasts the "degenerate single-step" MDPs of conventional LLM RL with the temporally extended, partially observable MDPs of agentic RL, in which RL develops capabilities such as planning, tool use, memory, and self-improvement.^[48] Work on reward design and credit assignment also advanced: subproblem curriculum RL (SCRL), introduced in May 2026, converts partial progress on hard problems into verifiable learning signals, reporting average accuracy gains over GRPO of 4.1 points on Qwen3-4B-Base and 1.9 points on Qwen3-14B-Base across seven math benchmarks.^[49] In robotics, sim-to-real RL grew markedly cheaper: a December 2025 recipe from Amazon's Frontier AI & Robotics (FAR) group trained humanoid locomotion policies for Unitree G1 and Booster T1 robots in about 15 minutes on a single RTX 4090 GPU, using the off-policy algorithms FastSAC and FastTD3.^[50]

Development tools and frameworks

Simulation environments

Frequently asked questions

Who invented reinforcement learning? No single person invented RL; it grew from three traditions (animal-learning psychology, optimal control, and temporal difference learning) that came together in the 1980s, largely through the work of Richard Sutton and Andrew Barto, whose 1998 textbook became the field's standard reference. The two shared the 2024 Turing Award for these foundations.^[7][51] Christopher Watkins introduced Q-learning in 1989,^[16] and Richard Bellman supplied the dynamic-programming mathematics in the 1950s.^[10]

Is reinforcement learning supervised or unsupervised? Neither. RL is its own paradigm, the third alongside supervised and unsupervised learning, because it learns from an evaluative reward signal rather than from labels or from unlabeled structure.^[1][2]

Is reinforcement learning used in ChatGPT? Yes. ChatGPT and other modern assistants are aligned using Reinforcement Learning from Human Feedback (RLHF), and reasoning models such as DeepSeek-R1 are trained with RL using verifiable rewards.^[31][33]

What is the most popular RL algorithm? Proximal Policy Optimization (PPO), published by OpenAI in 2017, is one of the most widely used general-purpose algorithms; OpenAI called it its default RL algorithm, and it was the optimizer used for InstructGPT's RLHF.^[24][31][64] For LLM reasoning, Group Relative Policy Optimization (GRPO) and its variants have become common.^[33][52]

Continue reading

• OpenAIOpenAI is an American artificial intelligence research and deployment organization.Mentioned 26 times above
• Q-LearningQ-learning is a model-free, off-policy reinforcement learning algorithm that learns the value of taking a given action in a given state by iteratively updating an action-value function Q(s, a), the expected…Mentioned 24 times above
• DDeep Q-Network (DQN)Deep Q-Network (DQN) is a reinforcement learning algorithm that uses a deep neural network to approximate the optimal action-value function (Q-function)Mentioned 23 times above
• Reinforcement Learning from Human Feedback (RLHF)Reinforcement learning from human feedback (RLHF) is a family of methods in which judgments supplied by people are converted into a reward signal and a policy is optimized with reinforcement learning against…Mentioned 17 times above
• AlphaGoAlphaGo is a computer program developed by DeepMind that plays the board game Go, and it was the first program to defeat a human professional Go player on a full 19x19 board without a handicap.Mentioned 14 times above
• Supervised LearningSupervised learning is a machine learning paradigm in which a model is fitted to examples that pair an input with an observed target.Mentioned 11 times above

More from See also

• Group Sequence Policy Optimization (GSPO)
• Instruction backtranslation (Humpback)
• Depth up-scaling (DUS)
• Selective Language Modeling (Rho-1)
• Machine Learning
• Deep Learning
• Unsupervised learning
• Markov Decision Process (MDP)
• Google DeepMind
• Neural Network
• AI Alignment

Topics:Artificial IntelligenceDeep LearningMachine LearningReinforcement Learning

External links

• Reinforcement Learning: An Introduction (Sutton and Barto's textbook, free online)
• OpenAI Spinning Up (educational resource for deep RL)
• Gymnasium (standard RL environment library)
• Stable-Baselines3 Documentation
• Ray RLlib Documentation
• DeepMind Research
• RLHF Book by Nathan Lambert (comprehensive guide to RLHF)

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16Sutton, R. S., & Barto, A. G. (2018). *Reinforcement Learning: An Introduction* (2nd ed.). MIT Press.
2. ^1 ^2 ^3Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). "Reinforcement learning: A survey." *Journal of Artificial Intelligence Research*, 4, 237-285.
3. ^1 ^2Sutton, R. S., & Barto, A. G. (1998). *Reinforcement Learning: An Introduction* (1st ed.). MIT Press.
4. ^1 ^2Bellman, R. (1957). *Dynamic Programming*. Princeton University Press.
5. ^1 ^2Silver, D., et al. (2016). "Mastering the game of Go with deep neural networks and tree search." *Nature*, 529(7587), 484-489.
6. ^1 ^2 ^3 ^4 ^5OpenAI. (2019). "OpenAI Five defeats Dota 2 world champions." OpenAI Blog.
7. ^1 ^2 ^3ACM. (2024). "ACM A.M. Turing Award recognizes pioneers of reinforcement learning."
8. ^Pavlov, I. P. (1927). *Conditioned Reflexes: An Investigation of the Physiological Activity of the Cerebral Cortex*. Oxford University Press.
9. ^Thorndike, E. L. (1911). *Animal Intelligence: Experimental Studies*. Macmillan.
10. ^1 ^2Bellman, R. (1957). "A Markovian decision process." *Journal of Mathematics and Mechanics*, 6(5), 679-684.
11. ^1 ^2Sutton, R. S. (1988). "Learning to predict by the methods of temporal differences." *Machine Learning*, 3(1), 9-44.
12. ^Sutton, R. S., & Barto, A. G. (1998). *Reinforcement Learning: An Introduction*. MIT Press.
13. ^Puterman, M. L. (1994). *Markov Decision Processes: Discrete Stochastic Dynamic Programming*. Wiley.
14. ^Sutton, R. S. (1990). "Integrated architectures for learning, planning, and reacting based on approximating dynamic programming." *Proceedings of the 7th International Conference on Machine Learning*.
15. ^1 ^2Hafner, D., et al. (2023). "Mastering diverse domains through world models." *arXiv:2301.04104*. arxiv.org/...2301.04104. Peer-reviewed version: Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025). "Mastering diverse control tasks through world models." *Nature*, 640, 647-653.
16. ^1 ^2 ^3Watkins, C. J. C. H. (1989). *Learning from Delayed Rewards*. PhD thesis, University of Cambridge. Convergence proof in Watkins, C. J. C. H., & Dayan, P. (1992). "Q-learning." *Machine Learning*, 8(3), 279-292.
17. ^Rummery, G. A., & Niranjan, M. (1994). "On-line Q-learning using connectionist systems." *Technical Report CUED/F-INFENG/TR 166*, Cambridge University.
18. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Mnih, V., et al. (2015). "Human-level control through deep reinforcement learning." *Nature*, 518(7540), 529-533.
19. ^Williams, R. J. (1992). "Simple statistical gradient-following algorithms for connectionist reinforcement learning." *Machine Learning*, 8(3), 229-256.
20. ^Konda, V. R., & Tsitsiklis, J. N. (2000). "Actor-critic algorithms." *Advances in Neural Information Processing Systems*, 12.
21. ^1 ^2Mnih, V., et al. (2016). "Asynchronous methods for deep reinforcement learning." *Proceedings of the 33rd International Conference on Machine Learning*.
22. ^Lillicrap, T. P., et al. (2015). "Continuous control with deep reinforcement learning." *arXiv:1509.02971*.
23. ^Fujimoto, S., van Hoof, H., & Meger, D. (2018). "Addressing function approximation error in actor-critic methods." *Proceedings of the 35th International Conference on Machine Learning*.
24. ^1 ^2Schulman, J., et al. (2017). "Proximal policy optimization algorithms." *arXiv:1707.06347*.
25. ^Haarnoja, T., et al. (2018). "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." *Proceedings of the 35th International Conference on Machine Learning*.
26. ^1 ^2Tesauro, G. (1995). "Temporal difference learning and TD-Gammon." *Communications of the ACM*, 38(3), 58-68.
27. ^1 ^2 ^3Silver, D., et al. (2017). "Mastering the game of Go without human knowledge." *Nature*, 550(7676), 354-359.
28. ^1 ^2Silver, D., et al. (2018). "A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play." *Science*, 362(6419), 1140-1144.
29. ^1 ^2Vinyals, O., et al. (2019). "Grandmaster level in StarCraft II using multi-agent reinforcement learning." *Nature*, 575(7782), 350-354.
30. ^Christiano, P. F., et al. (2017). "Deep reinforcement learning from human preferences." *Advances in Neural Information Processing Systems*, 30.
31. ^1 ^2 ^3 ^4 ^5 ^6Ouyang, L., et al. (2022). "Training language models to follow instructions with human feedback." *Advances in Neural Information Processing Systems*, 35.
32. ^1 ^2 ^3Bai, Y., et al. (2022). "Constitutional AI: Harmlessness from AI feedback." *arXiv:2212.08073*.
33. ^1 ^2 ^3 ^4 ^5 ^6 ^7DeepSeek-AI. (2025). "DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning." *arXiv:2501.12948*.
34. ^Brown, N., & Sandholm, T. (2019). "Superhuman AI for multiplayer poker." *Science*, 365(6456), 885-890.
35. ^FAIR et al. (2022). "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." *Science*, 378(6624), 1067-1074.
36. ^OpenAI et al. (2019). "Solving Rubik's Cube with a robot hand." *arXiv:1910.07113*.
37. ^Komorowski, M., et al. (2018). "The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care." *Nature Medicine*, 24(11), 1716-1720.
38. ^Evans, R., & Gao, J. (2016). "DeepMind AI reduces Google data centre cooling bill by 40%." DeepMind Blog.
39. ^Busoniu, L., Babuska, R., & De Schutter, B. (2008). "A comprehensive survey of multiagent reinforcement learning." *IEEE Transactions on Systems, Man, and Cybernetics*, 38(2), 156-172.
40. ^Albrecht, S. V., Christianos, F., & Schäfer, L. (2024). *Multi-Agent Reinforcement Learning: Foundations and Modern Approaches*. MIT Press.
41. ^Dulac-Arnold, G., et al. (2019). "Challenges of real-world reinforcement learning." *arXiv:1904.12901*.
42. ^Amodei, D., et al. (2016). "Concrete problems in AI safety." *arXiv:1606.06565*.
43. ^Zhao, W., et al. (2020). "Sim-to-real transfer in deep reinforcement learning for robotics: A survey." *arXiv:2009.13303*.
44. ^Levine, S., et al. (2020). "Offline reinforcement learning: Tutorial, review, and perspectives on open problems." *arXiv:2005.01643*.
45. ^Reed, S., et al. (2022). "A generalist agent." *arXiv:2205.06175*.
46. ^DeepSeek-AI. (2025). "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning." *Nature*, 645(8081), 633-638. nature.com/...s41586-025-09422-z
47. ^DeepSeek-AI. (2026). "DeepSeek-V4-Pro model card." Hugging Face. huggingface.co/...DeepSeek-V4-Pro
48. ^Zhang, G., et al. (2025-2026). "The landscape of agentic reinforcement learning for LLMs: A survey." *arXiv:2509.02547*. arxiv.org/...2509.02547
49. ^Jiang, X., Tang, Z., Lin, W., Yue, Y., Wang, S., & Huang, G. "From reasoning chains to verifiable subproblems: Curriculum reinforcement learning enables credit assignment for LLM reasoning." (2026). *arXiv:2605.22074*. arxiv.org/...2605.22074
50. ^Seo, Y., Sferrazza, C., Chen, J., Shi, G., Duan, R., & Abbeel, P. (2025). "Learning sim-to-real humanoid locomotion in 15 minutes." *arXiv:2512.01996*. arxiv.org/...2512.01996
51. ^1 ^2 ^3 ^4 ^5ACM. (2025). "ACM Announces 2024 A.M. Turing Award Recipients: Andrew Barto and Richard Sutton." Association for Computing Machinery. The award carries a 1 million US dollar prize sponsored by Google. acm.org/...turing-award-2024 ; award citation (textbook "cited over 75,000 times"): awards.acm.org/...2024-turing
52. ^1 ^2 ^3 ^4 ^5Shao, Z., et al. (2024). "DeepSeekMath: Pushing the limits of mathematical reasoning in open language models." *arXiv:2402.03300*. (Introduces Group Relative Policy Optimization, GRPO.) arxiv.org/...2402.03300
53. ^1 ^2Lightman, H., Kosaraju, V., Burda, Y., et al. (2023). "Let's verify step by step." *arXiv:2305.20050*. (OpenAI; process vs outcome supervision and the PRM800K dataset.) arxiv.org/...2305.20050
54. ^1 ^2Lambert, N., et al. (2024). "Tulu 3: Pushing frontiers in open language model post-training." *arXiv:2411.15124*. (Allen Institute for AI; coins "reinforcement learning with verifiable rewards.") arxiv.org/...2411.15124
55. ^1 ^2 ^3Yu, Q., et al. (ByteDance Seed). (2025). "DAPO: An open-source LLM reinforcement learning system at scale." *arXiv:2503.14476*. arxiv.org/...2503.14476
56. ^Zheng, C., et al. (Qwen Team). (2025). "Group Sequence Policy Optimization." *arXiv:2507.18071*. arxiv.org/...2507.18071
57. ^1 ^2 ^3Hubert, T., et al. (AlphaProof and AlphaGeometry teams, Google DeepMind). (2025). "Olympiad-level formal mathematical reasoning with reinforcement learning." *Nature*, 651(8106), 607-613 (published online November 12, 2025). nature.com/...s41586-025-09833-y
58. ^1 ^2Google DeepMind. "AlphaGo." deepmind.google/...alphago
59. ^Silver, D., Hubert, T., Schrittwieser, J., et al. (2017). "Mastering chess and shogi by self-play with a general reinforcement learning algorithm." *arXiv:1712.01815*. arxiv.org/...1712.01815
60. ^1 ^2 ^3Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2013). "Playing Atari with deep reinforcement learning." NIPS Deep Learning Workshop. *arXiv:1312.5602*. arxiv.org/...1312.5602
61. ^Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017). "Building machines that learn and think like people." *Behavioral and Brain Sciences*, 40, e253. arxiv.org/...1604.00289
62. ^1 ^2 ^3 ^4Berner, C., et al. (OpenAI). (2019). "Dota 2 with large scale deep reinforcement learning." *arXiv:1912.06680*. arxiv.org/...1912.06680
63. ^OpenAI. (2018, June 25). "OpenAI Five." OpenAI Blog. web.archive.org/...openai-five
64. ^1 ^2OpenAI. (2017, July 20). "Proximal Policy Optimization." OpenAI Blog. web.archive.org/...openai-baselines-ppo
65. ^Haarnoja, T., et al. (2018). "Soft actor-critic algorithms and applications." *arXiv:1812.05905*. arxiv.org/...1812.05905
66. ^1 ^2Schrittwieser, J., et al. (2020). "Mastering Atari, Go, chess and shogi by planning with a learned model." *Nature*, 588(7839), 604-609. arxiv.org/...1911.08265
67. ^1 ^2 ^3Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., & Lin, M. (2025). "Understanding R1-Zero-like training: A critical perspective." Conference on Language Modeling (COLM 2025). *arXiv:2503.20783*. arxiv.org/...2503.20783
68. ^Nature. (2025, September 17). "Bring us your LLMs: why peer review is good for AI models." Editorial. nature.com/...d41586-025-02979-9
69. ^Gamble, C., & Gao, J. (2018, August 17). "Safety-first AI for autonomous data centre cooling and industrial control." DeepMind Blog. deepmind.google/...-cooling-and-industrial-control
70. ^Brohan, A., et al. (2023). "RT-2: Vision-language-action models transfer web knowledge to robotic control." *arXiv:2307.15818*. arxiv.org/...2307.15818
71. ^Wu, J., Yin, S., Feng, N., & Long, M. (2025). "RLVR-World: Training world models with reinforcement learning." *arXiv:2505.13934*. arxiv.org/...2505.13934
72. ^Fu, J., Zhao, X., Yao, C., Wang, H., Han, Q., Xiao, Y., & Wang, Y. (2025). "Reward shaping to mitigate reward hacking in RLHF." *arXiv:2502.18770*. arxiv.org/...2502.18770
73. ^Lu, Y., Fu, J., Tucker, G., et al. (2022). "Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios." *arXiv:2212.11419*. arxiv.org/...2212.11419
74. ^Chen, M., Beutel, A., Covington, P., Jain, S., Belletti, F., & Chi, E. (2018). "Top-K off-policy correction for a REINFORCE recommender system." *arXiv:1812.02353*. arxiv.org/...1812.02353
75. ^Rafailov, R., et al. (2023). "Direct preference optimization: Your language model is secretly a reward model." *arXiv:2305.18290*. arxiv.org/...2305.18290
76. ^Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024). "KTO: Model alignment as prospect theoretic optimization." *arXiv:2402.01306*. arxiv.org/...2402.01306
77. ^1 ^2 ^3 ^4OpenAI. (2022, November 30). "Introducing ChatGPT." OpenAI Blog. openai.com/...chatgpt
78. ^Clark, J., & Amodei, D. (2016, December 21). "Faulty reward functions in the wild." OpenAI Blog. web.archive.org/...faulty-reward-functions
79. ^DeepSeek. (2026, April 24). "DeepSeek V4 Preview Release." DeepSeek API Docs. api-docs.deepseek.com/...news260424
80. ^Boston Dynamics. (2024, March 19). "Starting on the right foot with reinforcement learning." bostondynamics.com/...-with-reinforcement-learning

Show all 80 references