Agent57

Agent57 is a model-free distributed reinforcement learning algorithm developed by Google DeepMind and reported in 2020. It was designed as an Atari-playing AI agent for the 57-game benchmark, with particular attention to games where earlier agents struggled with sparse rewards or long delays between actions and their consequences. In the authors' experiments, Agent57 was the first single deep reinforcement-learning algorithm to exceed the benchmark's standard average-human reference score in every one of the 57 games.^[1]

Agent57 builds directly on Recurrent Replay Distributed DQN (R2D2) and Never Give Up (NGU). It combines NGU's intrinsic-reward system and family of exploration policies with three principal changes: separate action-value estimates for extrinsic and intrinsic rewards, a bandit-based meta-controller that chooses among exploration and discount settings, and a longer recurrent training sequence.^[1] The result should not be confused with MuZero, which uses a learned model and tree search, or with a single multi-game checkpoint. Agent57 used the same algorithm and hyperparameter setting across games, but each Atari game had its own training runs.^[1][14]

The reported result is specific to the Arcade Learning Environment and its standard human-normalized metric. It does not show that Agent57 broadly matched human intelligence, learned with human-like efficiency, surpassed expert players, or transferred to unseen games. The last game to cross the paper's average-human reference, Skiing, required 78 billion frames of experience.^[1]

Publication history

Scroll sideways for more →

The archival Agent57 paper is an ICML conference paper, not a Nature article. The nearby Nature publications in this research lineage include the 2015 DQN paper and the December 2020 MuZero paper.^[8][11]

Background and design goal

The Arcade Learning Environment, introduced in 2013, exposed agents to Atari 2600 games through a common interface. Its creators presented it as a test of domain-independent competence because the games differ in their visual structure, reward timing, control requirements, and need for exploration.^[6] A typical algorithm receives screen pixels and game rewards, selects joystick actions, and is evaluated across many games without game-specific rules or handcrafted features.

Earlier Atari results often summarized performance with a mean or median across games. These aggregates can conceal a long lower tail. An algorithm may obtain extremely high scores on several games while learning little on a few difficult ones. Before Agent57, the Agent57 authors counted R2D2 as above the standard human reference on 52 games and MuZero on 51, even though both had high aggregate scores.^[1] The Agent57 work therefore focused on the number and lower percentiles of games above the reference, rather than treating mean score alone as evidence of broad coverage.

The paper identifies two recurring problems in the lower tail. Hard-exploration games such as Montezuma's Revenge, Pitfall!, Private Eye, and Venture may require hundreds of actions before the agent reaches a positive reward. Skiing and Solaris stress long-term credit assignment because a decision can affect a reward received much later.^[1] A setting that encourages novelty can help with the first problem but interfere with exploitation. A very high discount factor can help with the second but destabilize learning on other games. Agent57 addresses this conflict by learning several policies and changing which of them supplies experience as training progresses.

Relationship to R2D2 and Never Give Up

R2D2 adapted recurrent Q-Learning to distributed experience replay. Many actor processes interact with independent copies of an environment and send recurrent transition sequences to a prioritized replay buffer. A central learner samples sequences from that buffer, updates a recurrent action-value network, and periodically sends its weights back to the actors. R2D2 was intended to preserve recurrent state across replayed experience while gaining the throughput of distributed data collection.^[5]

NGU added directed exploration to that foundation. Its intrinsic reward combines short-term novelty within an episode and slower-changing novelty over the agent's training history. The episodic component compares a learned embedding of the current observation with nearby embeddings in an episodic memory. An inverse-dynamics task trains the embedding so that it emphasizes aspects of the observation related to the agent's actions. A Random Network Distillation predictor supplies the lifelong novelty component by measuring prediction error against a fixed random target network.^[4][10]

NGU multiplies the episodic novelty by a capped lifelong modifier and adds the resulting intrinsic reward to the game's extrinsic reward. Rather than learn only one policy, it conditions a shared recurrent network on a family of intrinsic-reward scales and discount factors. A larger intrinsic-reward coefficient favors exploration. A smaller coefficient favors the game's external score. The discount factor changes how strongly the policy values distant rewards.^[4]

This policy family improved hard exploration, but NGU assigned actors to its policies uniformly. Each policy therefore collected the same amount of experience regardless of whether it was useful for the current game or stage of training. The Agent57 paper also found that one conditional value network could become unstable when intrinsic and extrinsic rewards had very different scales and statistical properties.^[1]

Architecture and learning system

Separate intrinsic and extrinsic value functions

Agent57 divides the action-value estimate into extrinsic and intrinsic parts. In simplified notation, it selects actions using Q(x,a,j) = Qe(x,a,j) + beta_j * Qi(x,a,j), where j identifies one member of the policy family. Qe estimates value from the environment's reward, while Qi estimates value from the novelty reward. The two networks have the same architecture and see the same replayed sequences, but each has its own weights, optimizer state, reward stream, and transformed Retrace loss. They share the target policy implied by their combined value.^[1]

The separation does not change the intended combined objective under the paper's assumptions. It changes the parameterization. Each branch can adapt to the scale and variance of its own reward instead of forcing one conditional network to represent both. This matters when a novelty signal is dense and comparatively large while the game reward is sparse.^[1]

The networks retain the value transformation used in the R2D2 and NGU line. The transform compresses action values with h(x) = sign(x) * (sqrt(abs(x) + 1) - 1) + 0.001*x, reducing the range that the learner must represent. Agent57 uses transformed Retrace targets and losses, an off-policy multi-step method that corrects for differences between the actors' behavior and the target policy.^[1][9] For action selection, the published system used a simple linear mixture of the separately learned values. The supplement reports little difference between this identity mixture and a mixture formed after inverting the value transform.^[1]

The recurrent network consumes the current observation, previous action, previous extrinsic and intrinsic rewards, recurrent state, and a one-hot policy-family index. It uses LSTM recurrence and dueling value heads. The recurrent state lets the action-value estimate depend on a history of observations rather than only the current frame.^[1][5]

Policy family

Agent57 trains 32 policy settings. Each setting pairs an intrinsic-reward coefficient beta with a discount factor gamma. The reported coefficients run from a purely extrinsic policy with beta = 0 to a strongly exploratory policy with beta = 0.3. The discount choices span 0.99 to 0.9999. High-exploration settings generally use shorter horizons, while low-exploration settings include longer horizons suited to sparse or delayed external rewards.^[1]

These are policies inside one game-specific learner, not 32 separate Atari agents. They share training data and network structure, and the policy index conditions the value estimate. Training exploratory settings can function as an auxiliary task because they continue to visit novel states even when the game has not supplied a useful reward.^[1][4]

Adaptive meta-controller

Each actor runs its own non-stationary multi-armed-bandit meta-controller. At the start of an episode, the controller selects one of the 32 (beta, gamma) settings. The actor follows that setting for the whole episode, then supplies the undiscounted external return as the bandit's reward. Because the learner changes over time, the return distribution for any arm is also changing. Agent57 therefore uses a sliding-window form of Upper Confidence Bound with additional epsilon-greedy exploration rather than a stationary bandit rule.^[1]

Independent controllers are useful because Agent57's actors use different epsilon-greedy action rates. A policy setting that works well for a relatively random actor may not be the best setting for a more exploitative actor. The evaluator has a separate controller and periodically selects the arm with the highest recent empirical return.^[1]

The controller determines both exploration and effective planning horizon. On Skiing, it favored a high-discount setting once learning began. On Hero it generally preferred a larger novelty weight and shorter discount. For Gravitar, Crazy Climber, Beam Rider, and James Bond, the paper shows a shift from exploratory, shorter-horizon settings early in training toward more exploitative, longer-horizon settings later.^[1]

Distributed recurrent training

The distributed system uses 256 actors connected to a central prioritized replay buffer and a single GPU learner. The supplement reports about 260 environment steps per second for each actor and about five learner updates per second. Each learner update uses a minibatch of 64 sequences, each 160 steps long.^[1]

Actors initialize recurrent state at the start of an episode, choose a policy-family arm, interact epsilon-greedily, and attach initial replay priorities to the sequences they send. The learner updates those priorities from temporal-difference errors. A separate evaluator copies learner weights but does not add evaluation episodes to replay.^[1]

Agent57 doubled the main backpropagation-through-time sequence from R2D2's 80 steps to 160. This gives gradients a longer recurrent window over which to assign credit. It does not by itself make the full 78-billion-frame training history differentiable. Longer-term learning still depends on bootstrapped value targets, replay, recurrence, and the chosen discount factor.^[1][5]

Evaluation protocol

Agent57 was evaluated on the 57-game Atari suite using one architecture and hyperparameter setting across the games. The paper trained separate runs for each game and did not train one checkpoint jointly on all 57. For Agent57, the authors averaged evaluation returns across six seeds, smoothed them with a 50-episode window, and reported the maximum of that averaged curve over training. Most ablations used three seeds. MuZero values in the comparison came from the MuZero work rather than a new run by the Agent57 team.^[1]

The supplement specifies a 30-minute episode limit, four-frame action repeat, up to 30 random no-op actions at the beginning, grayscale input, max pooling over two frames, the full action set, and no terminal transition when a life is lost. It also states that sticky actions were disabled.^[1]

The main score is the human-normalized score:

HNS = (agent score - random score) / (human score - random score)

A value of 0 corresponds to the random-policy reference, and 1, often displayed as 100%, corresponds to the paper's average-human reference. Scores can be below 0 or far above 1. The paper also uses capped human-normalized score:

CHNS = max(min(HNS, 1), 0)

Capping makes each game contribute at most one unit to the aggregate. An enormous score on one game cannot compensate for failing another, so a capped mean of 100% requires every game to reach the reference.^[1]

Scroll sideways for more →

Reported results

In the final ICML proceedings table, Agent57 reached a capped mean of 100.00 and exceeded the standard human baseline on all 57 games. Its uncapped mean human-normalized score was 4,766.25%, its median was 1,933.49%, and its 5th percentile was 116.67%.^[1] These figures are author-reported evaluations under the protocol above.

Scroll sideways for more →

The table illustrates why the paper separated coverage from central tendency. R2D2 with a bandit controller and MuZero had higher mean or median figures in this comparison, but neither cleared the human reference on every game. Agent57's claim concerned the bottom of the distribution, not the highest average score.^[1]

Progress was highly uneven in sample count. Agent57 crossed the human baseline on 51 games within the first 5 billion frames. Montezuma's Revenge, Pitfall!, and Private Eye came later. Skiing crossed last, after 78 billion frames. In Skiing, lower scores are better: the paper lists random at -17,098.1, average human at -4,336.9, and the game's optimum at -3,272. Agent57's final score table gives -4,202.60 with a standard deviation of 607.85.^[1]

The difficult exploration games also had wide variation across seeds. The final table reports 9,352.01 plus or minus 2,939.78 on Montezuma's Revenge, 18,756.01 plus or minus 9,783.91 on Pitfall!, and 79,716.46 plus or minus 29,515.48 on Private Eye. Their average-human references were 4,753.3, 6,463.7, and 69,571.3 respectively.^[1] The large standard deviations are relevant when interpreting the threshold-crossing result.

Ablation studies

The principal ablations used a ten-game set containing Beam Rider, Freeway, Montezuma's Revenge, Pitfall!, Pong, Private Eye, Skiing, Solaris, Surround, and Venture. This combined six games previously identified as difficult exploration problems with games that stress delayed credit assignment.^[1]

Scroll sideways for more →

The random-coin experiment isolates why the reward split matters. The agent gets the external reward by reaching a coin quickly, but a strongly novelty-seeking policy benefits from avoiding the coin and visiting the rest of the room. When one conditional network represents both behaviors, the large intrinsic scale disrupts the externally rewarded policy. Separate value networks reduce that interference.^[1]

The bandit ablation also shows that Agent57's controller changes the effective horizon. When attached to R2D2 without intrinsic reward, it can choose among discount factors and improve long-horizon behavior. The paper presents this adaptive horizon as one reason the system can use gamma = 0.9999 where useful without imposing it on every game.^[1]

Compute, data, and reproducibility

Agent57's headline result required much more experience than earlier Atari evaluations that stopped at 200 million frames. Reaching 51 games within 5 billion frames and the final Skiing threshold at 78 billion frames makes the performance result inseparable from its training scale.^[1] A later paper from several Agent57 authors described the algorithm as requiring nearly 80 billion frames and used that figure as the baseline for a 200-fold sample-efficiency improvement.^[14]

The distributed throughput also used considerable parallelism. The supplement specifies 256 environment actors and a GPU learner, but it does not give a complete wall-clock, energy, or financial-cost accounting.^[1] The experience was generated by interaction rather than human demonstrations, yet the amount of interaction remained many orders of magnitude above what a person would need to become familiar with an Atari game.

The paper and supplement provide detailed formulas, network diagrams, preprocessing settings, and most hyperparameters. They also contain small specification conflicts. The main text gives one set of bandit window lengths, while the supplement's hyperparameter table lists a different bandit window. The implementation prose and hyperparameter table also differ on the actor weight-update period.^[1] The PMLR record links the paper and supplement but does not list an official implementation or trained checkpoints.

The evaluation protocol adds further qualifications. The supplement disables sticky actions even though the revised 2018 Arcade Learning Environment guidance recommended them as a standard way to introduce stochasticity and reduce exploitation of deterministic action sequences.^[7] Reporting the maximum over training also selects the best point on a noisy learning curve rather than a fixed final checkpoint. Later work on reinforcement-learning evaluation showed that few-run point estimates can have substantial uncertainty and that maximum-based protocols may not be comparable with end-of-training evaluations. That later analysis was broader than Agent57, but its cautions apply to reading the six-seed point estimates.^[13]

Interpretation of the human benchmark

The phrase above human in the Agent57 result refers to one historical average-human score for each game. It does not mean that the system beat the best human score or achieved a human level of general learning. Human normalization is a linear rescaling around a random agent and a reference player. It does not measure how quickly either learned, whether they used prior knowledge, or how well they adapted to new tasks.^[1][8][12]

Toromanoff, Wirbel, and Moutarde compared Atari agents with human world records and argued that the standard human baseline can understate attainable human performance. They calculated that world-record scores corresponded to a median of about 4,400% under the usual normalization, far above the 100% threshold.^[12] Agent57 itself makes a similar caution: its authors treated the baseline as a reference for reasonable performance, not as proof that Atari research was complete.^[1]

The system's breadth was also narrower than the phrase one agent can suggest. The implementation used the same algorithm and settings, but the model for one game was not a general checkpoint that could be placed into another. There was no demonstrated cross-game transfer, continual learning across the suite, zero-shot play, or adaptation to unseen games. The result established consistency across a fixed benchmark after extensive task-specific interaction.

Subsequent context

The 2021 Nature paper First return, then explore reported Go-Explore, a different approach to hard exploration. Go-Explore stores promising states, returns to them, and explores outward, then trains a robust policy. It reported large gains on Montezuma's Revenge and Pitfall! and released its code.^[15] Its mechanism is distinct from Agent57's novelty reward, recurrent value learning, and bandit-selected policy family.

In September 2022, several Agent57 authors released the preprint Human-level Atari 200x faster, later presented at ICLR 2023. Its Efficient Memory-based Exploration agent, commonly called MEME, began from Agent57 and reported clearing all 57 human baselines with about 390 million frames. The changes included an approximate trust-region method, normalization of losses and replay priorities, a deeper normalization-free network, and policy distillation.^[14] That work treated Agent57's sample inefficiency as the main problem to solve rather than disputing its benchmark result.

Later Atari reporting has placed more weight on fixed data budgets, confidence intervals, interquartile mean scores, and performance profiles across games.^[13] In that context, Agent57 remains an important 2020 result about lower-tail coverage under a large-scale distributed regime. Its evidence supports a specific claim: under the authors' Atari 57 protocol, the algorithm's reported score exceeded the average-human reference in every game. It does not support a broader claim that the system had achieved human intelligence.

References

1. ^Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell, *Agent57: Outperforming the Atari Human Benchmark*, Proceedings of the 37th International Conference on Machine Learning, PMLR 119:507-517, including supplementary material, July 2020. proceedings.mlr.press/...badia20a
2. ^Adrià Puigdomènech Badia et al., *Agent57: Outperforming the Atari Human Benchmark*, arXiv:2003.13350, submitted 30 March 2020. arxiv.org/...2003.13350
3. ^Google DeepMind, *Agent57: Outperforming the human Atari benchmark*, 31 March 2020. deepmind.google/...rming-the-human-atari-benchmark
4. ^Adrià Puigdomènech Badia et al., *Never Give Up: Learning Directed Exploration Strategies*, International Conference on Learning Representations, 2020. openreview.net/forum
5. ^Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney, *Recurrent Experience Replay in Distributed Reinforcement Learning*, International Conference on Learning Representations, 2019. openreview.net/forum
6. ^Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, *The Arcade Learning Environment: An Evaluation Platform for General Agents*, Journal of Artificial Intelligence Research 47, 253-279, 2013. doi.org/...jair.3912
7. ^Marlos C. Machado et al., *Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents*, Journal of Artificial Intelligence Research 61, 523-562, 2018. doi.org/...jair.5699
8. ^Volodymyr Mnih et al., *Human-level control through deep reinforcement learning*, Nature 518, 529-533, 2015. doi.org/...nature14236
9. ^Tobias Pohlen et al., *Observe and Look Further: Achieving Consistent Performance on Atari*, arXiv:1805.11593, 2018. arxiv.org/...1805.11593
10. ^Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov, *Exploration by Random Network Distillation*, International Conference on Learning Representations, 2019. openreview.net/forum
11. ^Julian Schrittwieser et al., *Mastering Atari, Go, chess and shogi by planning with a learned model*, Nature 588, 604-609, 2020. doi.org/...s41586-020-03051-4
12. ^Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde, *Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the Playing Field*, arXiv:1908.04683, 2019. arxiv.org/...1908.04683
13. ^Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare, *Deep Reinforcement Learning at the Edge of the Statistical Precipice*, Advances in Neural Information Processing Systems 34, 29304-29320, 2021. proceedings.neurips.cc/...cf475e7426eed5e-Abstract
14. ^Steven Kapturowski, Víctor Campos, Ray Jiang, Nemanja Rakićević, Hado van Hasselt, Charles Blundell, and Adrià Puigdomènech Badia, *Human-level Atari 200x faster*, International Conference on Learning Representations, 2023. openreview.net/forum and arxiv.org/...2209.07550
15. ^Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune, *First return, then explore*, Nature 590, 580-586, 2021. doi.org/...s41586-020-03157-9