# Actor-critic methods

> Source: https://aiwiki.ai/wiki/actor_critic
> Updated: 2026-07-24
> Fact-checked: 2026-07-24
> Categories: Algorithms, Deep Learning, Machine Learning, Reinforcement Learning
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Actor-critic methods." aiwiki.ai, 24 Jul 2026. https://aiwiki.ai/wiki/actor_critic
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

Actor-critic methods are a family of [reinforcement learning](https://aiwiki.ai/wiki/reinforcement_learning) algorithms that learn two things at once: a parameterized policy, called the actor, which chooses actions, and a value function, called the critic, which estimates how much future reward those choices are worth. The critic never selects an action. Its job is to score what the actor did, and that score sets the direction in which the policy parameters move. A Nature paper describing an actor-critic racing agent puts it plainly: the method "learns a policy (actor) that selects an action on the basis of the agent's observations and a value function (critic) that estimates the future rewards of each possible action" [25].

The design exists because the two obvious alternatives each have a weakness. Vijay Konda and John Tsitsiklis, whose convergence analysis is one of the standard references for the method, divide simulation-based methods into two groups. Actor-only methods estimate the gradient of performance with respect to the policy parameters directly by simulation, and their drawback is that "the gradient estimators may have a large variance"; because a new gradient is estimated independently of past estimates every time the policy changes, "there is no 'learning' in the sense of accumulation and consolidation of older information" [1]. Critic-only methods "rely exclusively on value function approximation" and are indirect, since "they do not try to optimize directly over a policy space", so a good value function does not by itself come with guarantees about the resulting policy [1]. Actor-critic methods "aim at combining the strong points of actor-only and critic-only methods": the critic uses an approximation architecture and simulation to learn a value function, which is then used to move the actor's parameters in a direction of performance improvement, holding out the promise of faster convergence than actor-only methods through variance reduction [1].

Many of the most heavily used deep reinforcement learning algorithms belong to this family, including [proximal policy optimization](https://aiwiki.ai/wiki/ppo), [deep deterministic policy gradient](https://aiwiki.ai/wiki/ddpg), [TD3](https://aiwiki.ai/wiki/td3) and [soft actor-critic](https://aiwiki.ai/wiki/soft_actor_critic). The pattern also carried over into [large language model](https://aiwiki.ai/wiki/large_language_model) post-training, where the PPO recipe used for [reinforcement learning from human feedback](https://aiwiki.ai/wiki/rlhf) pairs the language model (the actor) with a separately trained value model (the critic) [26].

## How the actor and the critic fit together

The setting is a [Markov decision process](https://aiwiki.ai/wiki/markov_decision_process_mdp): at each step the agent observes a state, picks an action, receives a reward, and moves to a new state. The actor is a policy with its own parameters, usually a neural network that outputs a probability distribution over actions (a stochastic policy) or a single action vector (a deterministic policy). The critic estimates either the state value, meaning the expected return from a state under the current policy, or the action value, meaning the expected return from taking a given action in that state.

The link between the two comes from the policy gradient theorem of Richard Sutton, David McAllester, Satinder Singh and Yishay Mansour, published at NIPS in 1999, which showed that the gradient of expected reward "can be written in a form suitable for estimation from experience aided by an approximate action-value or advantage function" [4]. That result is what licenses the actor to take its update direction from a learned critic rather than from raw sampled returns. Konda and Tsitsiklis derived a closely related insight in simultaneous and independent work, and their paper notes the overlap explicitly [1].

In practice the critic is trained by [temporal difference learning](https://aiwiki.ai/wiki/temporal_difference_learning): it compares its own prediction for a state against the reward actually received plus its prediction for the next state, and the mismatch, the temporal difference error, is used to correct it. The same error is what the actor consumes. Konda and Tsitsiklis describe their algorithms as "two-time-scale algorithms in which the critic uses TD learning with a linear approximation architecture and the actor is updated in an approximate gradient direction based on information provided by the critic" [5]. The two-time-scale structure matters: the critic has to track the value of a policy that is itself moving, so it is normally updated on a faster schedule than the actor.

Most modern implementations phrase the actor's update in terms of the advantage function, the difference between the value of taking a particular action in a state and the average value of that state. Subtracting a state-value baseline leaves the expected gradient unchanged while shrinking its variance, which is the core reason the critic earns its keep. A second Konda and Tsitsiklis result constrains what the critic should represent: because the actor has far fewer parameters than there are states, the critic need not approximate the full value function, only a projection of it onto a low-dimensional subspace whose basis functions are determined by the actor's own parameterization [1][5].

## Origins

Actor-critic architectures predate deep learning by three decades. Konda and Tsitsiklis attribute the approach to a 1983 paper by [Andrew Barto](https://aiwiki.ai/wiki/andrew_barto), [Richard Sutton](https://aiwiki.ai/wiki/richard_sutton) and Charles Anderson, "Neuronlike adaptive elements that can solve difficult learning control problems", published in IEEE Transactions on Systems, Man, and Cybernetics [1][2]. Barto and Sutton were named recipients of the 2024 ACM A.M. Turing Award, announced on 5 March 2025, for developing the conceptual and algorithmic foundations of reinforcement learning [31].

The actor-only branch of the family tree runs through Ronald Williams' 1992 paper on statistical gradient-following algorithms, which introduced the REINFORCE estimator [3]. The 1999 pairing of Sutton et al.'s policy gradient theorem with Konda and Tsitsiklis's actor-critic analysis is what turned the architecture into something with convergence arguments attached rather than a heuristic [4][5]. Konda and Tsitsiklis published the full treatment in SIAM Journal on Control and Optimization in 2003 [1].

Two later extensions matter for what followed. Jan Peters and Stefan Schaal's natural actor-critic, published in Neurocomputing in 2008, replaced the ordinary gradient with the natural gradient, which rescales the update by the geometry of the policy's parameter space [6]. Thomas Degris, Martha White and Richard Sutton then presented what they described as "the first actor-critic algorithm for off-policy reinforcement learning", online and incremental, with per-step cost linear in the number of learned weights [7].

## The deep learning era

The algorithm that made actor-critic methods central to deep RL was published by Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver and Koray Kavukcuoglu in February 2016 and presented at ICML that year. The paper proposed running many actor-learners in parallel and applying their gradients asynchronously, and reported that "parallel actor-learners have a stabilizing effect on training". Of the four algorithms it parallelized, the best was "an asynchronous variant of actor-critic", which "surpasses the current state-of-the-art on the [Atari](https://aiwiki.ai/wiki/atari) domain while training for half the time on a single multi-core CPU instead of a GPU" [8]. That variant is asynchronous advantage actor-critic, or A3C.

A2C is the synchronous version of the same idea. It was popularized through [OpenAI](https://aiwiki.ai/wiki/openai)'s Baselines library, whose A2C implementation cites the A3C paper as its source [19], and the [Stable-Baselines3](https://aiwiki.ai/wiki/stable_baselines) documentation describes A2C as "a synchronous, deterministic variant of Asynchronous Advantage Actor Critic (A3C)" that uses multiple parallel workers instead of a [replay buffer](https://aiwiki.ai/wiki/replay_buffer) [20]. In 2022 Shengyi Huang and co-authors showed that the separation between A2C and PPO is thinner than it looks: with other settings controlled, "A2C is a special case of PPO", and the two produce identical models [21].

Variance control on the actor's side came from generalized advantage estimation (GAE), introduced by [John Schulman](https://aiwiki.ai/wiki/john_schulman), Philipp Moritz, [Sergey Levine](https://aiwiki.ai/wiki/sergey_levine), Michael Jordan and [Pieter Abbeel](https://aiwiki.ai/wiki/pieter_abbeel) in June 2015. GAE uses "an exponentially-weighted estimator of the advantage function that is analogous to TD(lambda)" to cut the variance of policy gradient estimates "at the cost of some bias", with a trust region procedure applied to both the policy and the value function [9]. The trust region machinery it leaned on had been published a few months earlier as TRPO, an iterative procedure "with guaranteed monotonic improvement" that is similar to natural policy gradient methods [10]. In 2017 Schulman and coauthors at OpenAI simplified it into PPO, which kept "some of the benefits of trust region policy optimization (TRPO)" while being simpler to implement and better on sample complexity in practice [11]. PPO remains an actor-critic algorithm in structure: OpenAI's Spinning Up documentation notes that each iteration fits the value function "by regression on mean-squared error" alongside the clipped policy update, and that "PPO trains a stochastic policy in an on-policy way" [24].

| Algorithm | Introduced | Authors | Actor | Sampling regime |
|---|---|---|---|---|
| TRPO | 2015 | Schulman, Levine, Moritz, Jordan, Abbeel [10] | Stochastic | On-policy |
| DPG | 2014 | Silver, Lever, Heess, Degris, Wierstra, Riedmiller [12] | Deterministic | Off-policy |
| DDPG | 2015 | Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver, Wierstra [13] | Deterministic | Off-policy, replay buffer |
| A3C | 2016 | Mnih, Badia, Mirza, Graves, Lillicrap, Harley, Silver, Kavukcuoglu [8] | Stochastic | On-policy, asynchronous workers |
| A2C | Popularized via OpenAI Baselines [19][20] | OpenAI, after Mnih et al. | Stochastic | On-policy, synchronous workers |
| PPO | 2017 | Schulman, Wolski, Dhariwal, Radford, Klimov [11] | Stochastic | On-policy [24] |
| IMPALA | 2018 | Espeholt, Soyer, Munos, Simonyan, Mnih et al. [17] | Stochastic | Decoupled actors and learner, V-trace correction |
| TD3 | 2018 | Fujimoto, van Hoof, Meger [14] | Deterministic | Off-policy [23] |
| SAC | 2018 | Haarnoja, Zhou, Abbeel, Levine [15] | Stochastic | Off-policy [22] |

## Deterministic policy gradients, TD3 and SAC

[David Silver](https://aiwiki.ai/wiki/david_silver) and colleagues showed in 2014 that a deterministic policy has "a particularly appealing form" of gradient: it is the expected gradient of the action-value function, which "can be estimated much more efficiently than the usual stochastic policy gradient". To keep exploration alive they introduced an off-policy actor-critic algorithm that "learns a deterministic target policy from an exploratory behaviour policy", and reported that deterministic policy gradients beat their stochastic counterparts particularly in high-dimensional action spaces [12].

DDPG, published by Timothy Lillicrap and colleagues in September 2015, applied the ideas behind [deep Q-networks](https://aiwiki.ai/wiki/dqn) to continuous control on top of the deterministic policy gradient. Using the same learning algorithm, network architecture and hyperparameters throughout, it solved more than 20 simulated physics tasks including cartpole swing-up, dexterous manipulation, legged locomotion and car driving, and learned many of them end to end from raw pixel inputs [13].

DDPG's weak point is its critic. Scott Fujimoto, Herke van Hoof and David Meger showed in 2018 that the overestimation bias known from deep Q-learning "persists in an actor-critic setting", and proposed three corrections that together make up TD3: learning two critics and taking the minimum of the pair when forming the target, delaying policy updates relative to critic updates, and smoothing the target policy with noise [14]. Spinning Up describes the failure mode being fixed as a learned Q-function that "begins to dramatically overestimate Q-values, which then leads to the policy breaking, because it exploits the errors in the Q-function", and notes the paper's recommendation of one policy update for every two critic updates [23].

Soft actor-critic took the other route, published by Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine in January 2018. SAC is built on maximum [entropy](https://aiwiki.ai/wiki/entropy) reinforcement learning, in which "the actor aims to maximize expected reward while also maximizing entropy", that is, to succeed at the task while acting as randomly as possible. The paper reported state-of-the-art results on continuous control benchmarks and, unusually for an off-policy method at the time, very similar performance across random seeds [15]. A follow-up in December 2018 added a constrained formulation that "automatically tunes the temperature hyperparameter", removing the most awkward knob, and demonstrated the algorithm on real hardware including locomotion for a [quadrupedal robot](https://aiwiki.ai/wiki/quadruped_robot) and manipulation with a dexterous hand [16]. Spinning Up notes that SAC also adopted TD3's clipped double-Q trick, while its stochastic policy supplies the target smoothing effect for free [22].

## On-policy and off-policy actor-critic

The split that matters most in practice is whether the algorithm can reuse old data. On-policy actor-critic methods such as A2C, A3C and PPO learn only from trajectories generated by the current policy, so throughput comes from running many environments in parallel rather than from a replay buffer [20][24]. Off-policy methods such as DDPG, TD3 and SAC store transitions and replay them, which typically makes them more sample efficient, at the price of a critic that must evaluate a policy different from the one that produced the data [12][22][23].

Distributed training blurs the line. IMPALA, presented by Lasse Espeholt and colleagues in 2018, decouples acting from learning so that thousands of machines can generate experience while a central learner updates the parameters. Because the actors always lag behind the learner, the trajectories are slightly off-policy, which IMPALA corrects with "a novel off-policy correction method called V-trace" [17]. The same paper reported better performance with less data than previous agents on DMLab-30, a set of 30 tasks from the [DeepMind](https://aiwiki.ai/wiki/google_deepmind) Lab environment, and on Atari-57, along with positive transfer between tasks [17].

## Applications

Continuous control is where the family is strongest. TD3 was evaluated on the OpenAI [Gym](https://aiwiki.ai/wiki/gymnasium) task suite and reported better results than the prior state of the art in every environment tested [14], SAC reported state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods [15], and SAC's applications paper moved the algorithm onto physical robots [16].

The clearest large-scale demonstration outside a laboratory benchmark is Gran Turismo Sophy, the racing agent Sony AI reported in Nature in February 2022. The authors write that "we trained GT Sophy using a new deep RL algorithm we call quantile regression soft actor-critic (QR-SAC)", which extends SAC to handle N-step returns and replaces the expected value of future rewards with a representation of their probability distribution, training asynchronously from an experience replay buffer while actors keep driving with the latest policy [25]. GT Sophy won a head-to-head competition against four of the world's best Gran Turismo drivers [25].

Language model post-training inherited the same structure through PPO. The DeepSeekMath paper describes PPO as computing advantages from a learned value function using generalized advantage estimation, and notes the cost: "as the value function employed in PPO is typically another model of comparable size as the policy model, it brings a substantial memory and computational burden" [26]. That cost is the reason the same paper introduced group relative policy optimization (GRPO), which "foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources" [26].

## Limitations

The critic is both the source of the method's advantage and its main failure mode. A biased critic drags the actor toward a policy that exploits the critic's errors rather than the environment, which is exactly the overestimation pathology TD3 was built to suppress [14][23]. The [bias and variance tradeoff](https://aiwiki.ai/wiki/bias_variance_tradeoff) is explicit in GAE, which buys a large reduction in the variance of policy gradient estimates "at the cost of some bias" [9].

Reproducibility is a second recurring complaint. Peter Henderson and co-authors documented in 2017 how non-determinism in benchmark environments combines with variance intrinsic to the methods to make reported deep RL results hard to interpret, with random seeds and hyperparameter choices contributing heavily to the spread [18]. The original SAC paper framed its own contribution partly in these terms, listing "very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning" as the two problems limiting model-free deep RL in real domains [15].

A large empirical study posted in July 2026 put numbers on the [hyperparameter tuning](https://aiwiki.ai/wiki/hyperparameter_tuning) problem. Haseeb Shah, Lingwei Zhu, Adam White and Martha White analyzed more than 33,000 experiments on a single control task derived from a real water treatment plant, varying how the policy is updated, how it represents the distribution over actions, how its gradient is estimated, and how often it is updated relative to the value estimator. Their finding was that common defaults, Gaussian action distributions with pathwise gradient estimators, were among the least reliable configurations, while bounded distributions with adaptive update schedules stayed robust across a wide range of settings [30].

## Recent developments

A central argument about actor-critic methods in 2025 and 2026 has been whether language model training needs a critic at all. The [DeepSeek-R1](https://aiwiki.ai/wiki/deepseek_r1) paper states that "we adopt Group Relative Policy Optimization (GRPO), which foregoes the critic model that is typically the same size as the policy model", and the work was published in Nature in September 2025 [27]. [DAPO](https://aiwiki.ai/wiki/dapo), released in March 2025 by ByteDance Seed with the Institute for AI Industry Research at Tsinghua University and the University of Hong Kong, follows the same route: "compared to PPO, [GRPO](https://aiwiki.ai/wiki/grpo) eliminates the value function and estimates the advantage in a group-relative manner". DAPO reported 50 points on [AIME](https://aiwiki.ai/wiki/aime) 2024 using a [Qwen2.5](https://aiwiki.ai/wiki/qwen2_5)-32B base model, with the training code released on the verl framework [28].

The counter-argument arrived three weeks later from an overlapping author group. VAPO is an augmented proximal policy optimization framework for reasoning models that keeps the value model and attacks the reasons critics had been dropped, naming value model bias, heterogeneous sequence lengths and sparse reward signals as the problems to solve. Built on a Qwen 32B base model, VAPO reported a score of 60.4 on AIME 2024, more than 10 points ahead of both DeepSeek-R1-Zero-Qwen-32B and DAPO [29]. Neither position has settled the matter: dropping the critic cuts the memory and compute cost of training [26], while value-based frameworks report better scores on [mathematical reasoning](https://aiwiki.ai/wiki/mathematical_reasoning) benchmarks when the critic is engineered carefully [29].

## See also

- [Reinforcement learning](https://aiwiki.ai/wiki/reinforcement_learning)
- [Policy gradient methods](https://aiwiki.ai/wiki/policy_gradient)
- [Proximal policy optimization](https://aiwiki.ai/wiki/ppo)
- [Soft actor-critic](https://aiwiki.ai/wiki/soft_actor_critic)
- [TD3](https://aiwiki.ai/wiki/td3)
- [Reward model](https://aiwiki.ai/wiki/reward_model)

## References

1. Konda, V. R. and Tsitsiklis, J. N. "On Actor-Critic Algorithms." SIAM Journal on Control and Optimization, Vol. 42, No. 4, pp. 1143-1166, 2003. https://www.mit.edu/~jnt/Papers/J094-03-kon-actors.pdf
2. Barto, A. G., Sutton, R. S. and Anderson, C. W. "Neuronlike adaptive elements that can solve difficult learning control problems." IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(5), pp. 834-846, 1983. https://doi.org/10.1109/TSMC.1983.6313077
3. Williams, R. J. "Simple statistical gradient-following algorithms for connectionist reinforcement learning." Machine Learning, 8(3-4), pp. 229-256, 1992. https://doi.org/10.1007/BF00992696
4. Sutton, R. S., McAllester, D. A., Singh, S. P. and Mansour, Y. "Policy Gradient Methods for Reinforcement Learning with Function Approximation." Advances in Neural Information Processing Systems 12, 1999. https://proceedings.neurips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html
5. Konda, V. R. and Tsitsiklis, J. N. "Actor-Critic Algorithms." Advances in Neural Information Processing Systems 12, 1999. https://proceedings.neurips.cc/paper/1999/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html
6. Peters, J. and Schaal, S. "Natural Actor-Critic." Neurocomputing, 71(7-9), pp. 1180-1190, 2008. https://doi.org/10.1016/j.neucom.2007.11.026
7. Degris, T., White, M. and Sutton, R. S. "Off-Policy Actor-Critic." ICML 2012, arXiv:1205.4839. https://arxiv.org/abs/1205.4839
8. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D. and Kavukcuoglu, K. "Asynchronous Methods for Deep Reinforcement Learning." ICML 2016, arXiv:1602.01783 (4 February 2016). https://arxiv.org/abs/1602.01783
9. Schulman, J., Moritz, P., Levine, S., Jordan, M. and Abbeel, P. "High-Dimensional Continuous Control Using Generalized Advantage Estimation." arXiv:1506.02438 (8 June 2015). https://arxiv.org/abs/1506.02438
10. Schulman, J., Levine, S., Moritz, P., Jordan, M. I. and Abbeel, P. "Trust Region Policy Optimization." arXiv:1502.05477 (19 February 2015). https://arxiv.org/abs/1502.05477
11. Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O. "Proximal Policy Optimization Algorithms." arXiv:1707.06347 (20 July 2017). https://arxiv.org/abs/1707.06347
12. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D. and Riedmiller, M. "Deterministic Policy Gradient Algorithms." Proceedings of the 31st International Conference on Machine Learning, PMLR 32(1), 2014. https://proceedings.mlr.press/v32/silver14.html
13. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. and Wierstra, D. "Continuous control with deep reinforcement learning." arXiv:1509.02971 (9 September 2015). https://arxiv.org/abs/1509.02971
14. Fujimoto, S., van Hoof, H. and Meger, D. "Addressing Function Approximation Error in Actor-Critic Methods." ICML 2018, arXiv:1802.09477 (26 February 2018). https://arxiv.org/abs/1802.09477
15. Haarnoja, T., Zhou, A., Abbeel, P. and Levine, S. "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor." ICML 2018, arXiv:1801.01290 (4 January 2018). https://arxiv.org/abs/1801.01290
16. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P. and Levine, S. "Soft Actor-Critic Algorithms and Applications." arXiv:1812.05905 (13 December 2018). https://arxiv.org/abs/1812.05905
17. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S. and Kavukcuoglu, K. "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures." arXiv:1802.01561 (5 February 2018). https://arxiv.org/abs/1802.01561
18. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D. and Meger, D. "Deep Reinforcement Learning that Matters." AAAI 2018, arXiv:1709.06560. https://arxiv.org/abs/1709.06560
19. OpenAI Baselines. "A2C" module README. https://raw.githubusercontent.com/openai/baselines/master/baselines/a2c/README.md
20. Stable-Baselines3 documentation. "A2C." https://stable-baselines3.readthedocs.io/en/master/modules/a2c.html
21. Huang, S., Kanervisto, A., Raffin, A., Wang, W., Ontanon, S. and Dossa, R. F. J. "A2C is a special case of PPO." arXiv:2205.09123 (18 May 2022). https://arxiv.org/abs/2205.09123
22. OpenAI Spinning Up. "Soft Actor-Critic." https://spinningup.openai.com/en/latest/algorithms/sac.html
23. OpenAI Spinning Up. "Twin Delayed DDPG." https://spinningup.openai.com/en/latest/algorithms/td3.html
24. OpenAI Spinning Up. "Proximal Policy Optimization." https://spinningup.openai.com/en/latest/algorithms/ppo.html
25. Wurman, P. R. et al. "Outracing champion Gran Turismo drivers with deep reinforcement learning." Nature, Vol. 602, No. 7896, pp. 223-228, 9 February 2022 (DOI 10.1038/s41586-021-04357-7). https://www.cs.utexas.edu/~pstone/Papers/bib2html-links/nature22.pdf
26. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J. et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv:2402.03300 (5 February 2024). https://arxiv.org/abs/2402.03300
27. Guo, D., Yang, D., Zhang, H., Song, J., Wang, P. et al. "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning." Nature, Vol. 645, No. 8081, pp. 633-638, 17 September 2025 (DOI 10.1038/s41586-025-09422-z); preprint "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", arXiv:2501.12948. https://arxiv.org/abs/2501.12948
28. Yu, Q. et al. "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." arXiv:2503.14476 (18 March 2025). https://arxiv.org/abs/2503.14476
29. Yue, Y., Yuan, Y., Yu, Q., Zuo, X., Zhu, R. et al. "VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks." arXiv:2504.05118 (7 April 2025). https://arxiv.org/abs/2504.05118
30. Shah, H., Zhu, L., White, A. and White, M. "Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners." arXiv:2607.13274 (14 July 2026). https://arxiv.org/abs/2607.13274
31. U.S. National Science Foundation. "AI pioneers Andrew Barto, Richard Sutton win 2024 Turing Award." 5 March 2025. https://www.nsf.gov/news/ai-pioneers-andrew-barto-richard-sutton-win-2024-turing