OpenAI Five
OpenAI Five was a reinforcement learning system developed by OpenAI to play the competitive multiplayer video game Dota 2 at a professional level. On April 13, 2019, it beat Team OG, the reigning Dota 2 world champions, 2-0 in a best-of-three series in San Francisco.[1][3] OpenAI described it as "the first AI to beat the world champions in an esports game, having won two back-to-back games versus the world champion Dota 2 team, OG, at Finals this weekend."[3] The system was trained with Proximal Policy Optimization (PPO) in a single continuous run from June 30, 2018 to April 22, 2019. By OpenAI's count the Finals version had experienced about 45,000 years of Dota self-play over 10 real-time months, and at its peak the run averaged gradients across 1,536 GPUs.[1][3] Days after the OG match, OpenAI opened the bots to the public in the OpenAI Five Arena, where they won 99.4% of 7,257 competitive games (7,215 wins, 42 losses). OpenAI counted games that human teams abandoned as wins, and these accounted for 3,140 of the 7,215 wins.[1][10]
Background
Dota 2 is a multiplayer online battle arena (MOBA) game developed by Valve Corporation. In a standard match, two teams of five human players compete to destroy the opposing team's base structure, called the Ancient. Each player controls a "hero" with unique abilities; at the time of the OpenAI Five paper, players selected from a pool of 117 heroes.[1] Games run for approximately 45 minutes and require a combination of mechanical skill, strategic planning, team coordination, and real-time adaptation.[1]
OpenAI selected Dota 2 as a research challenge because the game presented several properties that made it far more difficult for AI systems than previous game-playing benchmarks like chess or Go.[2] OpenAI wrote that complex video games "start to capture the messiness and continuous nature of the real world."[2] Greg Brockman, OpenAI's co-founder and then-CTO, said in a statement that complex strategy games were a milestone for AI research "because they start to capture aspects of the real world."[21]
Why Was Dota 2 a Hard Problem for AI?
Dota 2 posed a number of challenges that distinguished it from earlier AI game-playing milestones:
Partial observability. Unlike chess or Go, where both players can see the entire board, Dota 2 uses a "fog of war" mechanic. Each team can only see the parts of the map near its own units and buildings, so much of the game state is hidden at any given time. Strong play requires making inferences from incomplete data and modeling the opponent's behavior.[1]
Long time horizons. Dota 2 runs at 30 frames per second, and OpenAI Five selected an action every fourth frame, which yields approximately 20,000 steps in a 45-minute game. By comparison, the paper notes, chess usually lasts 80 moves and Go 150.[1] Actions taken early in the game, such as resource allocation and lane assignments, can have consequences that only become apparent much later, which makes it hard to work out which actions led to a win or a loss.[1]
Enormous action space. OpenAI discretized the action space; on an average timestep, depending on the hero, the model chose among 8,000 to 80,000 actions. The factorized action space reached up to 1,837,080 dimensions in total.[1] For comparison, the paper gives a branching factor of around 35 valid actions for chess and around 250 for Go.[1]
High-dimensional observation space. OpenAI Five observed about 16,000 total values per timestep (mostly floats and categorical values) describing the game state, including unit positions, health values, ability cooldowns, and item inventories. Rather than processing raw screen pixels, OpenAI Five read structured data through Valve's bot API.[1][2]
Five-player coordination. Dota 2 is a team game requiring tight coordination among five players. Successful play demands role specialization, shared map control, coordinated team fights, and collective resource management. OpenAI's June 2018 post said the system did "not contain an explicit communication channel between the heroes' neural networks."[2]
Complex game mechanics. Dota 2's rules are implemented in hundreds of thousands of lines of code, with intricate interactions between hero abilities, items, terrain, and neutral creeps.[1] The game also receives frequent patches; OpenAI noted in 2018 that it gets "an update about once every two weeks."[2]
The 1v1 Bot (August 2017)
Development
Before tackling the full 5v5 game, OpenAI built a bot that played 1v1 mid-lane matches using the hero Shadow Fiend, the same hero Dendi played against it at The International 2017.[23] The bot learned the game from scratch through self-play, without imitation learning or tree search, by playing against copies of itself.[6] OpenAI had also been downloading every expert-level public replay since November 2016, building a dataset of 5.8 million games for possible behavioral cloning in the 5v5 game.[20]
OpenAI's published timeline shows rapid progress during 2017. On March 1, the system had its first classical reinforcement learning results in a simple Dota environment, in which a Drow Ranger learned to kite a hard-coded Earthshaker. In early June it beat a tester with a 1,500 matchmaking rating (MMR). By June 30 it was winning most games against a 3,000 MMR tester, and on July 8 it "barely" got its first win against a 7,500 MMR semi-professional tester.[20] OpenAI said that "in the span of a month" the system went from barely matching a high-ranked player to beating the top pros.[20]
The International 2017 Demonstration
On August 11, 2017, OpenAI staged a demonstration at The International 2017 (TI7), Dota 2's premier annual championship tournament, held in Seattle. The bot played Danylo "Dendi" Ishutin, a Ukrainian professional player and former world champion, on the main stage and won the best-of-three match 2-0.[6][20] OpenAI said the bot played under standard tournament rules, without AI-specific simplifications to the 1v1 format.[20]
In the days before the Dendi match, the bot also beat several other top players, according to OpenAI's timeline:[20]
| Date (2017) | Opponent | OpenAI's description | Result |
|---|---|---|---|
| August 7 | Blitz | 6.2k MMR former pro | Bot won 3-0 |
| August 7 | Pajkatt | 8.5k MMR pro | Bot won 2-1 |
| August 7 | CC&C | 8.9k MMR pro | Bot won 3-0 |
| August 9 | Arteezy | 10k MMR pro, top player | Bot won 10-0 |
| August 10 | SumaiL | 8.3k MMR pro, top 1v1 player | Bot won 6-0 |
| August 11 | Dendi | 7.3k MMR pro, former world champion | Bot won 2-0 |
Limitations
The 1v1 bot operated under significant constraints. It played only one hero (Shadow Fiend) in a 1v1 mid-lane format that eliminated most of the strategic complexity of the full game, with no teammates and no need for team coordination. OpenAI itself noted that the bot "can still be confused in situations very different from what it's seen": at a LAN event during TI7, players played more than 1,000 games against it and found exploits such as repeatedly pulling lane creeps away so that the bot's tower died through attrition.[20]
OpenAI Five: The 5v5 System
Architecture
OpenAI Five used a single policy, a recurrent neural network with approximately 159 million parameters, and ran five copies of it at once. In the paper's words, "Separate replicas of the same policy function (with identical parameters θ) are used to control each of the five heroes on the team."[1] Because vision is shared across a team in Dota 2, the replicas received nearly identical observations. They chose different actions because part of the input told each replica which of the five heroes it was controlling.[1] OpenAI's 2018 blog post had described the same setup as "a separate LSTM for each hero" and as a "team of five neural networks."[2]
The core of the network was a single-layer Long Short-Term Memory (LSTM) network with 4,096 units, which accounted for 84% of the model's total parameter count.[1] The LSTM had 1,024 units in the June 2018 version,[2] 2,048 units when the long training run began, and was doubled to 4,096 units on August 26, 2018.[1] The final combined policy and value network had 158,502,815 parameters.[1]
The observation space was processed into a single vector representing the game-state information available to a human player (unit positions, health, mana, cooldowns, items, and so on). Floating-point observations were normalized by subtracting a running mean and dividing by the standard deviation, then clipped to the range (-5, 5).[1]
The action space used a factored structure. At each timestep, a hero selected a primary action (from up to 30 possibilities; the number available averaged 8.1 in the games against OG), plus parameters for delay (4 dimensions), unit selection (189 dimensions), and spatial offset (81 dimensions). Multiplied together, this gave a theoretical action space of up to 1,837,080 dimensions. Hand-written action filters removed unavailable actions, for example abilities on cooldown or actions with no valid target nearby.[1]
OpenAI's June 2018 post said the system did "not contain an explicit communication channel between the heroes' neural networks."[2] The final network did include one point of shared processing: a "cross-hero pool" operation that max-pooled the first 25% of a vector across the five replicas, so that each could respond to the small set of observations that differed between heroes.[1] Beyond that, coordination came from training through self-play and from the team-spirit reward described below.[1][2]
How Was OpenAI Five Trained? PPO at Scale
OpenAI Five was trained using Proximal Policy Optimization (PPO), a policy gradient reinforcement learning algorithm developed at OpenAI, together with Generalized Advantage Estimation.[1] According to the paper, OpenAI considered other algorithms early in the project, including other policy gradient methods, Q-learning, and evolutionary strategies, and "PPO was the first to show initial learning progress."[1]
The training ran on a custom distributed training platform called "Rapid," hosted on Google Cloud Platform.[1] The hardware changed considerably over the project:
- In June 2018, OpenAI reported using 256 NVIDIA P100 GPUs and 128,000 preemptible CPU cores on Google Cloud, a snapshot of the system as of June 6, 2018.[2]
- During the main training run, the paper reports between 480 and 1,536 optimizer GPUs (which computed gradients and applied parameter updates), 500 to 1,440 GPUs running forward passes for the rollouts, and 80,000 to 172,800 rollout CPUs.[1]
The system was organized into several components:
| Component | Role |
|---|---|
| Rollout Workers (CPUs) | Simulated Dota 2 games and collected experience data |
| Forward Pass GPUs | Computed actions for rollout workers during gameplay |
| Optimizer GPUs | Sampled experience from the buffer, computed gradients, and updated model parameters |
| Controller | Distributed updated parameters to all components |
| Experience Buffer | Stored gameplay data for the optimizers to sample from |
Rollout workers and optimizers operated asynchronously. The system targeted a sample reuse ratio close to 1, meaning optimizers consumed experience data at roughly the same rate that rollout workers produced it. Stale data was found to be harmful, so the final design sent game data every 30 seconds of gameplay and updated to fresh parameters approximately once a minute.[1]
In June 2018, OpenAI said OpenAI Five played "180 years worth of games against itself every day."[2] By the OG match in April 2019, OpenAI said the Finals version had consumed 800 petaflop/s-days and experienced "about 45,000 years of Dota self-play over 10 realtime months," up from about 10,000 years over 1.5 months at The International 2018, for an average of 250 years of simulated experience per day.[3][12] The paper's own accounting puts the optimization compute at 770 plus or minus 50 PetaFLOP/s-days at the time of the OG match on April 13, 2019, and 820 plus or minus 50 PetaFLOP/s-days when training was turned off on April 22.[1] The paper cautions that this counts only the optimizer GPUs, which it estimates at roughly 30% of the run's cost in dollars.[1]
Each optimizer GPU computed gradients on minibatches of 120 samples of 16 timesteps each, and gradients were averaged across a pool that peaked at 1,536 GPUs, for an effective batch size of 2,949,120 timesteps at the peak; the paper describes the system as learning from batches of approximately 2 million frames every 2 seconds.[1] By the team's accounting, compared with AlphaGo the run used 50 to 150 times larger batches, a model roughly 20 times larger, and about 25 times longer training. The 180 days of actual training were spread across ten months of wall-clock time because of restarts and reverts.[1]
Reward Shaping
Training a reinforcement learning agent on a game as complex as Dota 2 with only a win/loss signal at the end of a 45-minute match would be extremely slow. OpenAI addressed this by designing a shaped reward function that gave intermediate rewards for actions "which humans playing the game generally agree to be good."[1] The team built the reward function once at the start of the project and made only minor tweaks as game versions changed.[1]
Selected entries from the paper's reward table:[1]
| Reward Component | Weight | Given to |
|---|---|---|
| Win | 5 | Team |
| Hero death | -1 | Solo |
| Gold gained (per unit) | 0.006 | Solo |
| Experience gained (per unit) | 0.002 | Solo |
| Health changed (fraction of max health) | 2 | Solo |
| Killed hero | -0.6 (offsets the large gold and experience reward) | Solo |
| Last hit | -0.16 (net reward about 0.4 after gold and experience) | Solo |
| Deny | 0.15 | Solo |
| Gained Aegis | 5 | Team |
| Tier 1 / Tier 2 / Tier 3 tower | 2.25 / 3 / 4.5 | Team |
| Barracks | 6 | Team |
Two further mechanisms modified the raw rewards:
-
Zero-sum adjustment. Each hero's reward was adjusted by subtracting the average reward of the enemy team, so that every reward was zero-sum.[1] OpenAI's 2018 post said this prevented the agents "from finding positive-sum situations."[2]
-
Exponential time weighting. All rewards other than the win/loss reward were multiplied by a factor that decays exponentially over the game (0.6 raised to the power of game time in units of 10 minutes). Without it, the learning procedure would focus on late-game moments, when heroes are far more powerful and rewards are larger in absolute terms.[1]
OpenAI tested sparse rewards on two systems. For the 1v1 bot, an experiment rewarding only winning or losing "trained an order of magnitude slower and somewhat plateaued in the middle."[2] For the 5v5 game, a small-scale ablation trained on win/loss alone still learned to beat a hand-coded scripted agent consistently, reaching a TrueSkill of 155, but at "a large penalty to sample efficiency relative to the shaped reward baseline."[1]
Team Spirit
One of the more notable design choices in OpenAI Five was a hyperparameter called "team spirit," denoted by the Greek letter tau. This parameter controlled the balance between individual and collective reward for each hero, using a simple formula:[1]
effective_reward[i] = tau * mean(all_hero_rewards) + (1 - tau) * hero_reward[i]
At tau = 0, each hero received reward only for its own actions. At tau = 1, every reward was split equally among all five heroes.[1] The paper says the team ultimately cared about optimizing for tau = 1, but found that lower team spirit reduced gradient variance in early training, giving agents a clearer reward for improving their individual mechanical and tactical skills.[1]
According to the paper's hyperparameter table, team spirit was raised from 0.3 to 1.0 over the course of OpenAI Five's training (and from 0.3 to 0.8 in the later Rerun experiment).[1] OpenAI's June 2018 blog post had described annealing it "from 0 to 1 over training."[2]
Self-Play and Opponent Sampling
OpenAI Five was trained without any human gameplay data, through self-play.[1] The one exception the paper notes is item builds: randomized item purchases during training were sampled from the distribution of items heroes usually buy in human games.[1] The system played 80% of its games against the latest version of itself and 20% against past versions. This mixture helped prevent "strategy collapse," in which the agent forgets how to play against a wide variety of opponents because it only needs a narrow set of strategies to beat its immediate predecessor.[1]
Past opponents were selected using a dynamic sampling system. Each past version had a quality score, opponents were sampled through a softmax over those scores, and a past version's score was lowered each time the current agent beat it.[1]
Surgery: Adapting to Change
Over the 296-day training run, OpenAI needed to modify the model many times to accommodate game patches, new game mechanics, and architectural improvements. The team developed a set of tools it called "surgery" to handle these transitions without restarting training.[1]
Where possible, surgery produced a new model that implemented exactly the same function as the old one. When new observations were added, for example, the new input weights were initialized to zero so that the model's behavior was unchanged at first.[1] Changes to the environment or action space, such as upgrading to a new Dota 2 version or handing the buyback decision from scripted logic to the model, could not be made exactly equivalent. For these, the team "annealed" the change in, starting with 0% of rollout games using the new environment or action and slowly ramping up to 100%.[1]
In total, the team performed more than twenty successful surgeries, along with many unsuccessful attempts, roughly one every two weeks.[1] Documented changes included switching the environment from multiple couriers to the standard single courier, doubling the LSTM from 2,048 to 4,096 units, adding the items Bottle and Divine Rapier, handing the previously scripted buyback decision to the model, and absorbing the Dota 2 patches 7.19, 7.20, and 7.21.[1] The final environment change, to game version 7.21d, came eight days before the match against OG, which the team said would not have been possible if it had needed to retrain from scratch.[1]
Rerun: Validating the Final Pipeline
After the competitive phase ended, OpenAI trained a second agent, called "Rerun," from scratch on the final environment, model architecture, and codebase, starting on May 18, 2019.[1] Because it skipped the project's long history of game patches, rule changes, and architectural revisions, Rerun took two months and 150 plus or minus 5 PetaFLOP/s-days of compute, roughly 20 percent of the resources consumed by OpenAI Five itself.[1] Rerun kept improving past the champion-beating agent and reached a win rate of over 98 percent against the final version of OpenAI Five. Training was stopped at that point because the goal of validating the final code and hyperparameters had been met.[1] The team estimated that naively retraining from scratch after each of its roughly twenty major surgeries would have stretched the ten-month project to about 40 months.[1] On the project's internal TrueSkill scale, where 0 corresponds to random play, the version of OpenAI Five that defeated OG was rated 254.[1]
Timeline of Key Events and Matches
| Date | Event | Result |
|---|---|---|
| March 1, 2017 | First RL results in simplified Dota environment[20] | N/A |
| August 11, 2017 | 1v1 bot vs. Dendi at TI7[6] | Bot wins 2-0 |
| June 25, 2018 | OpenAI Five announced; beats amateur teams[2] | OpenAI Five wins |
| June 30, 2018 | Long training run of OpenAI Five begins[1] | N/A |
| August 5, 2018 | OpenAI Five Benchmark vs. a team of 99.95th-percentile players[5][1] | OpenAI Five wins 2-1 |
| August 22-23, 2018 | TI8 showmatches vs. paiN Gaming and a team of Chinese former stars[4] | OpenAI Five loses both games |
| April 13, 2019 | OpenAI Five Finals vs. OG (TI8 world champions)[3] | OpenAI Five wins 2-0 |
| April 18-21, 2019 | OpenAI Five Arena (public online event)[1] | 99.4% of 7,257 games won (3,140 of the 7,215 wins were abandoned games counted as wins) |
| April 22, 2019 | Training of OpenAI Five turned off[1] | N/A |
| May 2019 | "Rerun" experiment retrains a fresh agent on the final environment[1] | Exceeds 98% win rate vs. final OpenAI Five |
The paper also lists private evaluation series against professional teams between the TI8 losses and the OG match. OpenAI Five beat Team Lithium 3-0 (October 2018), SG Esports 4-0 (January 2019), and Alliance 3-0 (February 2019). It had lost two of three games to Team Secret in a private evaluation in August 2018.[1]
Game Restrictions
OpenAI Five played under restrictions that simplified the full Dota 2 game. Most were lifted over the project's lifetime, but some remained in place through the final matches against OG.
Early Restrictions (June 2018)
When OpenAI announced OpenAI Five in June 2018, it played only a mirror match of five fixed heroes (Necrophos, Sniper, Viper, Crystal Maiden, and Lich), with the following additional restrictions:[2]
- No warding and no Roshan (a powerful neutral boss).
- No invisibility (consumables and relevant items).
- No summons or illusions.
- No Divine Rapier, Bottle, Quelling Blade, Boots of Travel, Tome of Knowledge, or Infused Raindrop.
- Five invulnerable couriers, one per hero, with no exploiting them by scouting or tanking.
- No Scan.
Restrictions Removed Over Time
By the August 2018 Benchmark, OpenAI had lifted the restrictions on wards, Roshan, invisibility, and the fixed mirror match, expanded the hero pool to 18 heroes, and allowed every previously banned item except Divine Rapier and Bottle.[5] Summons and illusions, Scan, and the five invulnerable couriers were still excluded at the Benchmark.[5] OpenAI switched to a single mortal team courier on August 18, 2018, a few days before The International 2018, and later called the extra couriers "a very impactful restriction."[1][4] Bottle was added to the training environment on October 12, 2018 and Divine Rapier on November 28, 2018.[1]
In the published system description, OpenAI characterized the final version as playing with only two limitations relative to the regular game:[1]
- A 17-hero pool. Players in the normal game chose from 117 heroes; OpenAI Five supported 17 of them.[1] The pool had been 18 heroes at the Benchmark, and OpenAI removed Lich for the Finals "because his abilities were changed significantly in Dota version 7.20."[3][5] The 17-hero roster consisted of Axe, Crystal Maiden, Death Prophet, Earthshaker, Gyrocopter, Lion, Necrophos, Queen of Pain, Razor, Riki, Shadow Fiend, Slark, Sniper, Sven, Tidehunter, Viper, and Witch Doctor.[10]
- No items that let a player control multiple units at once. The Illusion Rune, Helm of the Dominator, Manta Style, and Necronomicon were removed to avoid the technical complexity of multi-unit control.[1]
Some game mechanics were also controlled by hand-scripted logic rather than the learned policy: the order in which heroes bought items and abilities, control of the courier, and which items heroes kept in their active inventory.[1]
The restricted hero pool was one of the most commonly cited limitations. Hero selection ("drafting") is a fundamental part of competitive Dota 2, and with only 17 heroes available this dimension of the game was severely limited. After The International 2018, OpenAI spent several weeks training with hero pools of up to 25 heroes, bringing them to approximately 5k MMR (about the 95th percentile of players). OpenAI said they "weren't learning fast enough to reach pro level before Finals."[3]
Ablation experiments published with the project paper suggest the restriction may have been less fundamental than it appeared. In early training, runs with an 80-hero pool progressed only about 20 percent slower than the base 17-hero runs, and the team hypothesized that an agent trained on the larger pool with Rerun's compute would reach a similar skill level with about 20 percent more training time. The paper cautions that the experiment covered only the very early stages of training and that the slowdown could become worse later on.[1]
How Did OpenAI Five Beat OG? The OpenAI Five Finals
The Event
On April 13, 2019, OpenAI hosted the "OpenAI Five Finals" in San Francisco. The headline match pitted OpenAI Five against OG, the winners of The International 2018 (TI8) and the reigning Dota 2 world champions at the time.[3] The event was broadcast on Twitch, with William "Blitz" Lee, Jorien "Sheever" van der Heijden, and Kevin "Purge" Godec on the desk and Austin "Capitalist" Walsh and Owen "ODPixel" Davies commentating.[9][1] The match was played on Dota 2 version 7.21d.[1]
OG's roster for the event included their full TI8-winning lineup:[1][9]
| Player | Real Name | Nationality | Position |
|---|---|---|---|
| ana | Anathan Pham | Australia | Carry (Position 1) |
| Topson | Topias Taavitsainen | Finland | Mid (Position 2) |
| 7ckngMad (Ceb) | Sebastien Debs | France | Offlane (Position 3) |
| JerAx | Jesse Vainikka | Finland | Support (Position 4) |
| N0tail | Johan Sundstein | Denmark | Support (Position 5) |
Match Results
OpenAI Five won both games, which OpenAI summarized as winning "two back-to-back games versus the world champion Dota 2 team, OG."[3]
Game 1 lasted 38 minutes and 18 seconds.[1] OpenAI Five, playing on the Radiant side, drafted Sniper, Gyrocopter, Crystal Maiden, Death Prophet, and Sven. OG played Earthshaker, Witch Doctor, Viper, Riki, and Shadow Fiend.[9] During the broadcast, OpenAI Five announced a 64.3 percent win estimate after OG's first pick (Earthshaker) and 67.6 percent by the end of the draft.[11] At around the five-minute mark OG killed several of OpenAI Five's heroes, lowering its estimate. At roughly 18 minutes OpenAI Five killed three human heroes in a row, regrouped in the middle lane, and declared a 95 percent probability of victory.[1]
Game 2 lasted 20 minutes and 51 seconds.[1] OpenAI Five drafted Crystal Maiden, Gyrocopter, Sven, Witch Doctor, and Viper. OG picked Sniper, Earthshaker, Death Prophet, Slark, and Lion.[9]
OpenAI noted that both it and DeepMind's AlphaStar had previously beaten professionals privately but lost their live professional matches, "making this also the first time an AI has beaten esports pros on livestream."[3]
The OpenAI Five Arena
Following the OG match, OpenAI opened the system to the public through the "OpenAI Five Arena," an online event that ran from April 18 to April 21, 2019. Anyone could form a team and play OpenAI Five in competitive or cooperative modes.[3] The competitive games used the same 17-hero pool as the Finals.[10]
Over the four-day period:
- OpenAI Five played 3,193 teams in 7,257 competitive games, winning 7,215 and losing 42, a win rate of 99.4%.[1]
- 3,140 of the 7,215 wins came from games that human teams abandoned before completion, sometimes immediately after an unfavorable draft. Abandoned games were counted as wins because OpenAI Five itself never abandons a game; the paper notes the figure likely includes a small number of games abandoned for technical or personal reasons.[1]
- Twenty-nine teams managed to defeat OpenAI Five, for a total of 42 games lost.[1] One team won 10 games.[21]
- The first human win came after 459 games, according to GamesBeat's summary of OpenAI's statistics.[22]
- OpenAI also reported 35,466 cooperative games, and a total of 10.7 years of game time across competitive and cooperative play.[21]
Sources differ on how many people took part. OpenAI's results announcement gave the "number of human players" as 30,937.[21] The paper's acknowledgements thank "all 39,356 Dota 2 players from 225 countries who participated in OpenAI Five Arena," the figure GamesBeat also reported.[1][22] Liquipedia, citing the archived Arena results page, gives 15,019 players who played competitively.[10]
Cooperative Play with Humans
Although OpenAI Five was trained purely to win at self-play, the Finals event also included a cooperative game. Blitz and Sheever played together with three OpenAI Five agents against ODPixel and Capitalist, who played with three agents controlled by a separate copy of OpenAI Five.[3] The Arena offered a cooperative mode alongside the competitive one, letting the public play with the agents as teammates rather than only against them.[3][13] Sheever said her bot-controlled Viper teammate "gave his life for me at some point," adding: "But, you know, he believed in me."[3] OpenAI described this as zero-shot transfer: the system "was trained to have all heroes controlled by copies of itself, but generalizes to controlling a subset of heroes, playing with or against humans."[3]
How Did OpenAI Five Compare to AlphaStar?
Around the same period, DeepMind developed AlphaStar, an AI system for Blizzard Entertainment's real-time strategy game StarCraft II.[8] The two projects are among the most prominent examples of applying deep reinforcement learning to complex competitive video games. They shared broad similarities but differed substantially in their technical approaches.
| Attribute | OpenAI Five | AlphaStar |
|---|---|---|
| Game | Dota 2 (MOBA) | StarCraft II (RTS) |
| Developer | OpenAI | DeepMind |
| Year of peak result | 2019 | 2019 |
| Players per team | 5 heroes, each controlled by a replica of one network | 1 AI agent |
| Core architecture | Single-layer 4,096-unit LSTM | Self-attention unit encoder, deep LSTM core, auto-regressive policy with pointer network |
| Model parameters | ~159 million | 139 million (55 million needed at inference) |
| Training method | Self-play with PPO | Supervised learning on human replays, then RL with league-based self-play |
| Human data used | None for gameplay (item-build randomization drew on human purchase statistics) | 971,000 human replays |
| Compute hardware | 480-1,536 optimizer GPUs plus rollout GPUs and 80,000-172,800 CPUs (main run) | 32 TPUv3s per agent, 12 agents in the league |
| Training duration | ~10 months wall-clock (180 days of training) | 44 days (league training) |
| Game experience | ~45,000 years equivalent (OpenAI estimate) | Not directly comparable (league-based) |
| Input representation | Structured game-state vectors (bot API) | Structured game-state data via a camera-like interface (no raw pixels) |
| Game restrictions | 17-hero pool, no multi-unit-control items | Full game, with action-rate limits and a camera-like view |
| Peak achievement | Defeated TI8 champions OG 2-0 | Grandmaster level for all three races, above 99.8% of ranked players |
| Publication | arXiv, December 2019 | Nature, October 2019 |
One notable difference was the use of human data. AlphaStar bootstrapped its training with supervised learning on 971,000 human replays before moving to reinforcement learning, while OpenAI Five learned to play from scratch through self-play.[8][1] AlphaStar also used a "league" training approach in which multiple agents, including dedicated exploiter agents, were trained against each other, whereas OpenAI Five trained a single agent against its current and past selves.[8][1]
Another key distinction involved game restrictions. AlphaStar played the full StarCraft II game with all three races, interacting through a camera-like interface and with action-rate limits meant to approximate human constraints.[8] OpenAI Five never played with the full hero roster.[1]
Technical Contributions and Impact
Scaling Reinforcement Learning
OpenAI Five showed that a well-known reinforcement learning algorithm, scaled to very large amounts of compute, could reach the top level of a highly complex game. The paper describes the key ingredient as scaling "existing reinforcement learning systems to unprecedented levels."[1] OpenAI wrote that it had expected to need sophisticated algorithmic ideas such as hierarchical reinforcement learning, but "the fundamental improvement we needed for this problem was scale."[3] The system did not use tree search, though some game mechanics (item and ability purchases, courier control, and item swapping) were handled by hand-scripted logic.[1]
OpenAI researcher Jakub Pachocki and chief executive Sam Altman later described the result as an early lesson in the importance of scaling (see below).[17][19]
Transfer to Robotics
OpenAI used the same reinforcement learning algorithms and training code as OpenAI Five for Dactyl, a 2018 project that trained a robotic hand (a Shadow Dexterous Hand) in simulation to reorient a block or other object in its palm.[7] Dactyl ran on the same Rapid training system, using 6,144 CPU cores and 8 GPUs, and collected about one hundred years of experience in 50 hours.[7] OpenAI pointed to Dactyl as evidence that the approach could control a physical robot even though training happened entirely in simulation.[3][7]
Emergent Coordination
One of the more surprising findings was the degree of team coordination that emerged without an explicit communication channel. OpenAI's 2018 post quoted Blitz: "The teamwork aspect of the bot was just overwhelming. It feels like five selfless players that know a good general strategy."[2] Over the course of training, the agents learned to group up for large team fights and to concentrate resources on their strongest heroes, a common human strategy. Compared with human players, they moved heroes back and forth across the map more often.[1]
Reaction Time and Mechanical Skill
OpenAI Five observed and acted on every fourth frame, an effective rate of 7.5 observations and actions per second. Combined with a one-step delay in issuing actions, this gave the agent a reaction time randomly distributed between 5 and 8 frames (167 to 267 milliseconds).[1] The paper says frameskip was used "to speed up our game execution and in order to bring reactions of our model closer to the human scale."[1] It reports an average reaction time of 217 milliseconds, against a typical human visual reaction time of roughly 250 milliseconds measured in controlled settings.[1] Before the August 2018 Benchmark, OpenAI had raised the bot's reaction time from 80 milliseconds to 200 milliseconds.[5]
Limitations Acknowledged
OpenAI was open about the system's limitations. The restricted hero pool meant that OpenAI Five never experienced the full strategic complexity of Dota 2's drafting phase.[1] At The International 2018, OpenAI Five had a good chance of winning for the first 20-35 minutes of both games before losing. OpenAI said the paiN Gaming game ended with the bot "succumbing to various high-level strategic pushes by human players," and that it needed "more training, bugfixes, and to remove the last pieces of scripted logic."[4] During the Arena, players reported weaknesses such as poor defense against split-pushing and against invisible heroes, and several high-level teams were able to win semi-consistently toward the end of the event.[10]
The computational cost also raised questions about sample efficiency. The tens of thousands of years of simulated gameplay were far beyond what any human player would need to reach a similar skill level, and OpenAI wrote that "decreasing the amount of experience is a next challenge for RL."[3]
Retirement and Legacy
On April 15, 2019, OpenAI said it was "retiring OpenAI Five as a competitor."[3] After the Arena concluded on April 21, 2019, training was turned off on April 22, 2019.[1] The team published its findings in a paper titled "Dota 2 with Large Scale Deep Reinforcement Learning" on arXiv on December 13, 2019.[1]
The project showed that self-play reinforcement learning could produce coordinated team behavior in a complex, partially observable environment, and that PPO could be run stably at very large batch sizes. It also contributed to a growing body of evidence that compute scaling could unlock capabilities that had previously seemed to require fundamental algorithmic breakthroughs.[1][3]
OpenAI Five also introduced a broad gaming audience to modern AI research through its matches at TI7, TI8, and the OpenAI Five Finals. According to GamesBeat, 486,000 Twitch users watched OpenAI Five Arena games.[22] For many in the gaming community, the matches against Dendi and OG were tangible demonstrations of how far machine learning had progressed.
Later Developments (2020-2026)
The line of work OpenAI Five opened continued to shape both games research and OpenAI's own trajectory after the project's retirement.
- MOBA AI after OpenAI Five. Researchers at Tencent AI Lab and Tencent TiMi Studio presented a system at NeurIPS 2020 that extended large-scale self-play reinforcement learning to the mobile MOBA Honor of Kings with a 40-hero pool. It won 95.2% of 42 matches against professionals.[14] The authors cited the fact that "OpenAI's Dota AI limits the play to a pool of only 17 heroes" as the main limitation their training paradigm addressed.[14]
- From game-playing to language models. Proximal Policy Optimization, the algorithm OpenAI Five ran at very large batch sizes, was later used as the policy-optimization method in reinforcement learning from human feedback (RLHF) for InstructGPT in 2022,[15] and OpenAI said ChatGPT was trained "using the same methods as InstructGPT, but with slight differences in the data collection setup."[16]
- The scaling thesis. In a 2025 blog post, Altman wrote that Pachocki and Szymon Sidor "decided to scale up RL as a baseline to see where it broke when the conventional wisdom was that it didn't scale which led to our Dota result," and credited the pair with building much of OpenAI's later infrastructure and leading GPT-4 pretraining.[17] Pachocki succeeded Ilya Sutskever as OpenAI's chief scientist in May 2024.[18] In a 2025 TIME 100 AI profile, Pachocki said the Dota 2 victory "was a big thing for OpenAI," adding that it meant "realizing that scaling was going to be very important."[19]
Continue reading
- OpenAIOpenAI is an American artificial intelligence research and deployment organization.Mentioned 137 times above
- Reinforcement learningReinforcement learning (RL) is a branch of machine learning in which an agent learns to make decisions by taking actions in an environment to maximize a cumulative reward signal, discovering good behavior…Mentioned 15 times above
- Proximal Policy Optimization (PPO)Proximal Policy Optimization (PPO) is an on-policy policy gradient reinforcement learning algorithm that stabilizes training by clipping the policy update so the new policy stays close ("proximal") to the old…Mentioned 7 times above
- AlphaStarAlphaStar is an artificial intelligence system built by Google DeepMind that in 2019 became the first AI to reach Grandmaster level in the real-time strategy game StarCraft IIMentioned 7 times above
- Dactyl (OpenAI)Dactyl was a robotics research project at OpenAI that used deep reinforcement learning to control a five-fingered, human-like robot hand and manipulate physical objects with high dexterity.Mentioned 3 times above
- Greg BrockmanGreg Brockman is an American software engineer and entrepreneur who co-founded OpenAI in December 2015 and serves as its president; he was chairman of the board until November 2023.Links back to this article
More to read
Topics:AI in GamingArtificial IntelligenceOpenAIReinforcement Learning
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49 ^50 ^51 ^52 ^53 ^54 ^55 ^56 ^57 ^58 ^59 ^60 ^61 ^62 ^63 ^64 ^65 ^66 ^67 ^68 ^69 ^70 ^71 ^72 ^73 ^74 ^75 ^76 ^77 ^78 ^79 ^80 ^81 ^82 ^83 ^84 ^85 ^86 ^87 ^88 ^89 ^90 ^91OpenAI. "Dota 2 with Large Scale Deep Reinforcement Learning." arXiv:1912.06680, December 2019. arxiv.org/...1912.06680
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17OpenAI. "OpenAI Five." OpenAI Blog, June 25, 2018. openai.com/...openai-five
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20OpenAI. "OpenAI Five defeats Dota 2 world champions." OpenAI Blog, April 15, 2019. openai.com/...-five-defeats-dota-2-world-champions
- ^1 ^2 ^3OpenAI. "The International 2018: Results." OpenAI Blog, August 23, 2018. openai.com/...the-international-2018-results
- ^1 ^2 ^3 ^4 ^5OpenAI. "OpenAI Five Benchmark." OpenAI Blog, July 18, 2018. openai.com/...openai-five-benchmark
- ^1 ^2 ^3OpenAI. "Dota 2." OpenAI Blog, August 11, 2017. openai.com/...dota-2
- ^1 ^2 ^3OpenAI. "Learning Dexterity." OpenAI Blog, July 2018. openai.com/...learning-dexterity
- ^1 ^2 ^3 ^4Vinyals, O. et al. "Grandmaster level in StarCraft II using multi-agent reinforcement learning." Nature 575, 350-354, October 2019. nature.com/...s41586-019-1724-z
- ^1 ^2 ^3 ^4Liquipedia. "OpenAI Five Finals." liquipedia.net/...OpenAI_Five_Finals
- ^1 ^2 ^3 ^4 ^5Liquipedia. "OpenAI Five Arena." liquipedia.net/...OpenAI_Five_Arena
- ^The Game Haus. "DOTA 2: OpenAI Five vs OG." May 9, 2019. thegamehaus.com/...09
- ^OpenAI. "How to Train Your OpenAI Five." OpenAI Blog, April 15, 2019 (the original URL of the post later retitled "OpenAI Five defeats Dota 2 world champions"; see reference 3). openai.com/...how-to-train-your-openai-five
- ^Coldewey, D. "OpenAI Five crushes Dota2 world champs, and soon you can lose to it too." TechCrunch, April 15, 2019. techcrunch.com/...-and-soon-you-can-lose-to-it-too
- ^1 ^2Ye, D. et al. "Towards Playing Full MOBA Games with Deep Reinforcement Learning." NeurIPS 2020; arXiv:2011.12692, November 2020. arxiv.org/...2011.12692
- ^Ouyang, L. et al. "Training language models to follow instructions with human feedback." arXiv:2203.02155, March 2022. arxiv.org/...2203.02155
- ^OpenAI. "Introducing ChatGPT." OpenAI Blog, November 30, 2022. openai.com/...chatgpt
- ^1 ^2Altman, S. "Jakub and Szymon." Sam Altman's blog, 2025. blog.samaltman.com/jakub-and-szymon
- ^OpenAI. "Ilya Sutskever to leave OpenAI, Jakub Pachocki announced as Chief Scientist." OpenAI Blog, May 14, 2024. openai.com/...achocki-announced-as-chief-scientist
- ^1 ^2Pillay, T. "Jakub Pachocki: The 100 Most Influential People in AI 2025." TIME, August 26, 2025. time.com/...jakub-pachocki
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8OpenAI. "More on Dota 2." OpenAI Blog, August 16, 2017. openai.com/...more-on-dota-2
- ^1 ^2 ^3 ^4Engadget. "OpenAI's 'Dota 2' bot won 7,215 games against humans in three days." Engadget, April 23, 2019. engadget.com/...3-openai-five-dota-2-arena-results
- ^1 ^2 ^3GamesBeat. "OpenAI's Dota 2 bot defeated 99.4% of players in public matches." GamesBeat, April 22, 2019. gamesbeat.com/...99-4-of-players-in-public-matches
- ^Chalk, Andy. "Elon Musk heralds the future of AI as a Dota 2 bot beats a human champ." PC Gamer, August 14, 2017. pcgamer.com/...as-a-dota-2-bot-beats-a-human-champ
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
12 revisions · v12 · 6,536 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independently verified 2026-10-30 against the OpenAI Five paper (arXiv:1912.06680), archived 2017-2019 OpenAI posts, Liquipedia, the AlphaStar Nature paper and contemporaneous press coverage; three minor issues corrected.
Cite this page: AI Wiki. "OpenAI Five." aiwiki.ai, updated 30 Oct 2026, fact-checked 30 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/openai_five


