Agent Arena
Agent Arena is a public leaderboard run by Arena (the evaluation company formerly called LMArena, which began as Chatbot Arena) that ranks large language models by how well they perform as the orchestrator of an AI agent during real user sessions. It launched on June 4, 2026, together with Arena's Agent Mode, a product in which users hand an agent multi-step work such as writing code, researching the web, building apps and producing documents.[1][5] Unlike Arena's older leaderboards, which turn pairwise human votes into ratings, Agent Arena does not ask users to compare two answers. It mines signals from single-agent sessions (explicit task confirmations, praise and complaints, corrections, shell errors and invented tool calls) and uses a randomized causal-inference method that Arena calls "causal tracing" to estimate each model's "net improvement" over the average model.[1][2] As of the October 16, 2026 snapshot, the board covered 52 models and 2,534,501 sessions, with Claude Opus 5.5 (Max) ranked first.[2]
Background
Arena traces its origins to Chatbot Arena, an academic project at the University of California, Berkeley. The company renamed itself from LMArena to Arena on January 28, 2026, and moved to the arena.ai domain.[20] Most of its other leaderboards are built from pairwise human votes, in which users see two anonymous responses and pick the better one, and Arena computes those ratings with Bradley-Terry regression.[6]
Agent Mode, announced on June 4, 2026, gives a model a sandbox with bash, web search, image generation, file upload and file writing, and lets it plan and execute a whole workflow from one prompt.[5][6] Arena said at launch that the data from these real-world tasks would "power a public agent leaderboard."[5] Arena described the resulting leaderboard as "real-world agentic evals at scale."[7]
Name
Several unrelated or older projects use similar names:
| Name | What it is | Relationship |
|---|---|---|
| Agent Arena (2024) | A platform from the Berkeley Gorilla team, published on the LMSYS/Arena blog on October 3, 2024, in which users compared two agents built from an LLM, a framework (such as LangChain, LlamaIndex or CrewAI) and tools, then voted for the better one | Same organization's blog and same name, but a different method: pairwise votes and an "extended" Bradley-Terry score over agent subcomponents[13] |
| Windows Agent Arena | A 2024 benchmark of multimodal agents operating Windows | Separate benchmark, not run by Arena[22] |
| Speech Agent Arena | A component used by Artificial Analysis in its speech-to-speech index | Separate evaluation, covered on the Artificial Analysis page |
The 2024 project's contributors included Shishir G. Patil, Wei-Lin Chiang, Anastasios N. Angelopoulos, Joseph Gonzalez and Ion Stoica. It routed a user's task to two agents with a GPT-4o-based router, streamed both agents' actions, and fitted ratings for whole agents and for individual models, tools and frameworks using L2-regularized logistic regression. It released 2,000 pairwise agent battles alongside the announcement.[13]
How it works
Sessions and random assignment
When a user starts an Agent Mode chat, Arena assigns a randomly chosen orchestrator model, the main model that decides which tools to call, and keeps it for the session.[2][6] Unlike Battle Mode, Agent Mode does not reveal which model was used, even after feedback is given; Arena says this keeps the focus on the work and supports fair evaluation.[6] Arena's help center also says that if a response is likely to fail and break the session, the system may switch to a different model so the session can continue.[6]
Arena's methodology post describes the agent as a multi-component system (orchestrator model, subagents, image models, tools, system prompt and harness), with each component choice treated as a possible "treatment." Randomizing those choices turns the live traffic into what Arena calls "a multi-intervention randomized controlled trial."[1] The leaderboards published so far evaluate only the orchestrator, so in production the number of randomized components is one.[1]
Causal tracing and net improvement
For each signal, Arena defines a model's treatment effect as the expected difference in outcome when the orchestrator is set to that model, relative to a fixed baseline distribution that is typically uniform over the models. It estimates this with a self-normalized importance-weighted estimator, reports a 95% confidence interval with every estimate, and applies time-decaying weights so that recent sessions count more as new models enter.[1] Arena calls the resulting contrast, expressed in percentage points, the "net improvement." Positive values mean better than the average model and negative values mean worse.[2]
Because every score is relative to the average model on the board, scores can move without any change in a model's behavior. Arena's leaderboard FAQ notes that adding a strong model raises the bar for everyone else and adding a weak one lowers it.[2] The page also shows a rank range for each model; on October 16, 2026, Claude Opus 5.5 (Max) had a rank range of 1 to 2 and GPT-6 Astra (Max) a range of 1 to 7.[2]
Signals
The headline leaderboard aggregates five signals. The definitions below are from Arena's signal-definitions page as last updated on October 12, 2026.[3]
| Signal | What is scored | Notes |
|---|---|---|
| Confirmed Success | Whether the user answers "yes" to a pop-up asking "Did this complete your task?" | No partial credit. Phrases such as "perfect, thanks" or a silent ending do not count as confirmation[3] |
| Praise vs Complaint | Whether a task contains more unprompted praise than complaints | Tasks with equal counts, including the common case of no explicit sentiment, are not scored[3] |
| Steerability | Steering effort per assistant turn: none, low (one correction that lands) or high (a correction that has to be repeated) | Redefined on September 30, 2026 to score every judged turn[3][4] |
| Bash Reliability | Agent-caused bash failures per call, across all non-exempt calls | Previously "Bash Recovery." Errors outside the agent's control, such as network outages, are exempt; tests that run correctly but find problems, and searches that find nothing, count as successes[3][4] |
| Tool Hallucination | Rate of calls to tools that do not exist, use a nonexistent signature, or claim a capability the agent lacks | Arena says base rates are low for most frontier models and the signal is most informative at the margin[3] |
The headline number is a weighted average of the five per-signal net improvements. As of October 16, 2026, the leaderboard FAQ states that Bash Reliability carries half the weight of each other signal.[2] Arena's help center, still unrevised when checked on October 19, 2026, describes the average as equally weighted, says the weighting may change and still lists Bash Recovery among the signals.[6] Arena says it also tracks behaviors that do not yet count toward the headline score, including clean continuation, disapproval, in-session retries and a tool-error rate.[2]
Cost, categories and views
Arena calculates the realized cost of each session. Its methodology post says some models turn out more expensive in practice than their list prices suggest, because they take more steps per turn or lead users to take more turns.[1] The leaderboard reports median (P50) cost per task and output tokens per task, offers a Pareto view of score against median cost over the last 14 days, and can be filtered by license (proprietary or open) and by category (Overall, Code, Chat and Work).[2][11]
Methodology changes
| Date | Change |
|---|---|
| June 4, 2026 | Leaderboard launches with five signals: confirmed success, praise vs. complaint, steerability, bash recovery and tool hallucination[1] |
| September 30, 2026 | Steerability rescored. The old version only measured whether a model fixed a mistake right after a correction, so a model that made more mistakes had more chances to "pass." The new version evaluates every judgeable assistant turn and still weights repeated corrections more heavily[4] |
| October 1, 2026 | Methodology update note added to the launch blog post, pointing to new signal definitions[1] |
| October 12, 2026 | Bash Recovery replaced by Bash Reliability (details below)[4] |
From Bash Recovery to Bash Reliability
Under the launch definition, the bash signal counted how many follow-up bash calls an agent needed after a model-caused error before it issued a non-erroring command, with an added penalty if the agent gave up.[1] Arena's October 12, 2026 changelog entry gave two reasons for replacing it. First, the rule-based failure detector flagged correct tests and empty searches as failures and missed errors hidden by piped commands; Arena now uses an LLM judge that looks at each call's purpose, outcome, cause of any error, and whether the error was fixed within the call. Second, the old score only looked at sessions where something went wrong, so "a model with flawless bash use wasn't being credited," and any later successful call counted as a "recovery" even when it was unrelated to the original failure. The new score measures failures per call, so a session with no bash failures counts in a model's favor.[4] Arena said many models shifted as a result and that backfilled scores would be published in its public Hugging Face datasets.[4]
AlphaSignal's coverage summarized the effect: under Bash Recovery, "a model that failed frequently could receive substantial credit by repairing its commands," while Bash Reliability assigns the original failure to the model even when a later call succeeds.[17] Three days after the change, Arena announced a new top five among open models "with this update."[11]
Timeline
| Date | Event |
|---|---|
| October 3, 2024 | Berkeley Gorilla team publishes the earlier, vote-based Agent Arena[13] |
| June 4, 2026 | Agent Mode and the Agent Arena leaderboard launch; the first snapshot is built from more than 300,000 tasks, 2 million tool calls and 40 million lines of agent-written code[5][7] |
| June 10, 2026 | Arena reports Claude Fable 5 at No. 1, leading on confirmed success and praise vs. complaint "despite weaker steerability"[8] |
| June 16, 2026 | GLM-5.2 (Max) enters at No. 10, which Arena calls the top open model "by a wide margin"[9] |
| September 30, 2026 | Steerability redefined[4] |
| October 2, 2026 | GPT-6.1 Sol (Max) enters at No. 5 with +11.23% at a $0.56 median cost per task; Arena lists Claude Fable 5.1 (Max) at No. 1[10] |
| October 8, 2026 | Arena announces a $200 million Series B at a $3.1 billion valuation and the Alignment Index, built from Agent Arena sessions; it reports 7 million Agent Arena sessions in under five months[14][15] |
| October 12, 2026 | Bash Recovery becomes Bash Reliability; Arena publishes results for its agentic Max router[4][16] |
| October 15, 2026 | MiMo V2.6 Pro and MiMo V2.6 Flash are added; Arena reports MiMo V2.6 Pro as the top open model[4][11] |
| October 16, 2026 | Reflection's Beam 1.0 501B A23B is added; snapshot covers 52 models and 2,534,501 sessions[2][4] |
Usage data
Arena's methodology post publishes statistics from a recent seven-day slice of Agent Mode traffic.[1]
| Measure (7-day slice) | Value |
|---|---|
| Agent Mode tasks | 160,480 (a session can contain several tasks) |
| Sessions | 128,244, of which 75.6% used at least one tool |
| Sessions running bash / web search | 41.1% / 27.1% |
| Structured tool calls | About 2 million, including about 936,000 bash calls, 550,000 file writes and 275,000 web searches |
| Lines of code written (successful write_file calls) | 40.3 million, roughly 1,000 per coding session |
| Largest task categories | Code writing 17.5%, research and lookup 10.8%, planning and brainstorming 10.6%, multimodal image/video 10.2%, document creation 9.1%, code debugging 8.9% |
| Workspace downloads | More than 50,000, including .docx, .pptx, .xlsx, .pdf and image files |
| Long-context sessions | About 32% ended with at least 128k input tokens in the final turn, 22% with at least 256k and 8% with at least 1M |
The same post names two failure patterns Arena observed in traces. "Bluffing" is when an agent presents incomplete work as complete instead of surfacing what is missing. "Bluster" is assertive pushback that "melts under additional pressure."[1] Arena also reported that most opening messages hand over a whole job, but that users then tighten control after seeing the first response.[1]
Leaderboard snapshots
All figures are Arena's own measurements. Because scores are relative to a moving average, numbers from different dates are not directly comparable.
Launch, June 4, 2026
| Rank | Model | Net improvement |
|---|---|---|
| 1 | GPT-5.5 (High) | +10.7% |
| 2 | Claude Opus 4.7 (Thinking) | +9.5% |
| 3 | GPT-5.4 (High) | +8.9% |
| 4 | Claude Opus 4.6 | +8.1% |
| 5 | GPT-5.5 | +7.5% |
| 6 | Claude Opus 4.7 | +7.0% |
| 7 | Claude Sonnet 4.6 | +4.6% |
| 8 | GLM-5.1 | +3.4% |
| 9 | Gemini-3.1 Pro | +1.4% |
| 10 | Gemini-3.5 Flash | +0.4% |
The launch chart listed 18 models; the lowest was Grok-4.3 at -25.1%.[7]
Open models, October 15, 2026
Arena's announcement after the Bash Reliability change listed this top five among open models:[11]
| Open-model rank | Model | Net improvement |
|---|---|---|
| 1 | MiMo V2.6 Pro | +3.8% |
| 2 | DeepSeek V4.1 Flash (Max) | +3.0% |
| 3 | Hy4 preview | +2.9% |
| 4 | MiMo V2.6 Flash | +2.5% |
| 5 | GLM-5.2 (Max) | +2.4% |
Arena said MiMo V2.6 Pro ($0.11 per task) and MiMo V2.6 Flash ($0.05 per task) were the only two open models on the Agent Arena Pareto frontier, and that the Pro model cost 15% less than DeepSeek V4.1 Flash and 71% less than Hy4 preview.[11] Xiaomi repeated the result in its own post the next day.[12] The accompanying chart ranked 18 open models, from MiMo V2.6 Pro at +3.8% down to Thinking Machines Lab's Inkling Small at -14.7%.[11]
Overall, October 16, 2026
The leaderboard snapshot dated October 16, 2026 (52 models, 2,534,501 sessions) showed the following top ten.[2]
| Rank | Model | Developer | Net improvement (95% CI) | Sessions | Median cost per task |
|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 (Max) | Anthropic | +18.88% (±2.74) | 10,551 | $6.00 |
| 2 | GPT-6 Astra (Max) | OpenAI | +14.13% (±2.30) | 13,168 | $3.76 |
| 3 | Claude Fable 5.1 (Max) | Anthropic | +14.03% (±1.99) | 16,374 | $7.44 |
| 4 | GPT-6.1 Sol (Max) | OpenAI | +13.60% (±1.90) | 13,444 | $0.72 |
| 5 | Gemini 4 Argon (High) | +13.23% (±1.33) | 30,489 | $1.01 | |
| 6 | Claude Sonnet 5.5 (Max) | Anthropic | +12.99% (±3.13) | 7,560 | $3.73 |
| 7 | GPT-6 Sol (Max) | OpenAI | +10.31% (±2.10) | 7,887 | $1.43 |
| 8 | Claude Opus 5 (Max) | Anthropic | +8.68% (±1.69) | 23,770 | $5.24 |
| 9 | Claude Fable 5 (High) | Anthropic | +8.18% (±1.42) | 42,608 | $3.26 |
| 10 | GPT-5.6 Sol (xHigh) | OpenAI | +7.08% (±1.41) | 36,545 | $1.56 |
Signal leaders in the same snapshot were Claude Sonnet 5.5 (Max) for Confirmed Success (+21.13%), Claude Opus 5.5 (Max) for Praise vs Complaint (+39.93%), Steerability (+16.40%) and Bash Reliability (+15.42%), and GPT-6.1 Sol (Max) for Tool Hallucination (+0.67%).[2] Bash Reliability separated models that were otherwise close: GPT-6.1 Sol (Max) scored +2.35% and Gemini 4 Argon (High) -2.48% on that signal, against +15.42% for Claude Opus 5.5 (Max).[2] The weakest Tool Hallucination score belonged to Mistral Large 4 at -17.46%, and the last-placed model overall was Upstage's Solar Pro 4 at -20.06%.[2]
The highest-ranked entries with a non-proprietary license on October 16 were:[2]
| Overall rank | Model | License listed | Net improvement | Median cost per task |
|---|---|---|---|---|
| 13 | MiMo-V2.6-Pro | MIT | +3.95% | $0.12 |
| 19 | DeepSeek-V4.1-Flash-20260910 (Max) | MIT | +2.74% | $0.14 |
| 20 | Hy4 preview | Apache 2.0 | +2.68% | $0.41 |
| 21 | MiMo-V2.6-Flash | MIT | +2.60% | $0.06 |
| 22 | GLM-5.2 (Max) | MIT | +2.26% | $0.92 |
| 23 | Kimi K3 (Max) | Kimi K3 license | +1.93% | $1.11 |
| 24 | GLM-5.3 (Max) | MIT | +1.78% | $0.88 |
| 30 | DeepSeek-V4-Pro (High, 20260813) | MIT | +0.16% | N/A |
Leaders over time
| Date | No. 1 overall | Net improvement | Source |
|---|---|---|---|
| June 4, 2026 | GPT-5.5 (High) | +10.7% | Arena launch post[7] |
| June 10, 2026 | Claude Fable 5 | Not stated in post | Arena[8] |
| June 29, 2026 | Claude Fable 5 (High) | 13.34% | Capital & Compute reading of the leaderboard[18] |
| October 2, 2026 | Claude Fable 5.1 (Max) | Not stated in post | Arena[10] |
| October 13, 2026 | Claude Opus 5.5 (Max) | 17.89% | AlphaSignal[17] |
| October 16, 2026 | Claude Opus 5.5 (Max) | +18.88% | Leaderboard snapshot[2] |
Related Arena products
Alignment Index. Announced on October 8, 2026, the index uses three signals drawn from Agent Arena conversations: Unauthorized Action (the model acts beyond what the user asked), False Attribution (it attributes something to the user that user-provided evidence contradicts) and Deceptive Completion (it reports a task as done when it is not). The first release compared 27 models across 90,000 sessions. Arena reported that OpenAI models held the top five positions and that about 10% of sessions on average were affected by a deceptive completion, rising to 48% in code debugging.[15] Arena describes the index as complementary to Agent Arena, which ranks capabilities.[14]
Max router. On October 12, 2026, Arena described a new version of its Max router, which it said is powered by more than 2.5 million Agent Arena sessions and routes each request according to task complexity, cost and task type. In the reported test it routed among Fable 5.1 (Max), Astra (Medium) and Astra (Max). Arena said that in a live test the router ranked No. 1 in Agent Arena with +14.7% at a median cost of $3.27 per task, 0.3 points above Fable 5.1 (Max) at 37.5% lower cost. The router was trained on data collected before September 17, before Claude Opus 5.5 was released.[16]
Reception and limitations
Agent Arena is an Arena product, and the figures above are Arena's own measurements. Labs republish them in their own announcements, as Xiaomi did with its MiMo result.[12] Arena itself flags several limits. The board ranks orchestrator models only. Its five signals are "only a starting point," and Arena says it will add, retire and modify them.[1] Scores are relative to a moving average, and overlapping confidence intervals mean close ranks should be read as ties.[2]
An explainer by Capital & Compute, which called Agent Arena "the most interesting agent leaderboard to appear in 2026," listed four caveats: the user population is self-selected, so tasks skew toward whatever arena.ai users do; the signal definitions change over time, which makes the board hard to track across months; confidence intervals below the top overlap heavily; and the Praise vs Complaint signal is still partly about style, so a model tuned to elicit warm language could raise its score "without doing better work." It also noted that Arena sells evaluations to the labs whose models appear on its boards.[18] Both definition changes so far (Steerability in September and Bash Reliability in October 2026) moved scores, which Arena acknowledged in its changelog.[4]
Writing about the Series B and the Alignment Index, Remio argued that Arena's evidence would carry more weight with detailed methodological disclosure and independent replication, since "a model that behaves well inside Agent Arena might perform differently with another harness" with different tool permissions, prompts and runtime constraints.[19] Arena's earlier vote-based leaderboards had faced criticism in the 2025 paper The Leaderboard Illusion, which argued that undisclosed private testing let a handful of providers test multiple variants before public release and retract scores. That critique concerned the Chatbot Arena voting system, not Agent Arena's causal method.[21]
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13Arena Team. "Agent Arena: Causal Evaluation of Agents in the Real World." Arena blog, published June 4, 2026, last updated October 15, 2026. arena.ai/...agent-arena-methodology
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17Arena. "Agent Arena Leaderboard" (overall view, snapshot dated October 16, 2026, including FAQ). Accessed October 19, 2026. arena.ai/...agent
- ^1 ^2 ^3 ^4 ^5 ^6Arena. "Agent Arena Signal Definitions." Last updated October 12, 2026. arena.ai/...agent-arena-signals
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11Arena. "Leaderboard Changelog." Entries from June 4 to October 16, 2026. Accessed October 19, 2026. arena.ai/...leaderboard-changelog
- ^1 ^2 ^3 ^4Arena Team. "Empowering Users to Get More Done With Agent Mode." Arena blog, June 4, 2026. arena.ai/...agent-mode
- ^1 ^2 ^3 ^4 ^5 ^6Arena Help Center. "How to use Agent Mode on Arena." Accessed October 19, 2026. help.arena.ai/...5432423882-how-to-use-agent-mode
- ^1 ^2 ^3 ^4Arena (@arena). "Introducing Agent Arena: real-world agentic evals at scale." Post on X with leaderboard chart, June 4, 2026. x.com/...2062566749418233981
- ^1 ^2Arena (@arena). "Exciting news: Claude Fable 5 ranks #1 on the new Agent Arena leaderboard!" Post on X, June 10, 2026. x.com/...2064807170714358193
- ^Arena (@arena). "GLM-5.2 (Max) by @Zai_org ranks #10 on the new Agent Arena leaderboard." Post on X, June 16, 2026. x.com/...2066943450914943025
- ^1 ^2Arena (@arena). "GPT-6.1 Sol (Max) by @OpenAi just landed in the Agent Arena at #5 (+11.23%)." Post on X, October 2, 2026. x.com/...2106109027923140928
- ^1 ^2 ^3 ^4 ^5 ^6Arena (@arena). "There's a new leader in the Agent Arena among open models." Post on X with charts, October 15, 2026. x.com/...2110815995950731340
- ^1 ^2Xiaomi MiMo (@XiaomiMiMo). "MiMo V2.6 Pro ranks #1 on the Agent Arena open-model leaderboard." Post on X, October 16, 2026. x.com/...2110944071875485925
- ^1 ^2 ^3Yekollu, N., Bohra, A., Chirumamilla, A., Wen, K., Kolasani, S., Chiang, W.-L., Angelopoulos, A. N., Gonzalez, J., Stoica, I., and Patil, S. G. "Agent Arena." Arena blog, published October 3, 2024. arena.ai/...agent-arena-2024
- ^1 ^2Arena Team. "Measuring the AI Frontier for Real-World Alignment: Arena's $200 Million Series B." Arena blog, October 8, 2026. arena.ai/...series-b
- ^1 ^2Arena Team. "Arena Alignment Index." Arena blog, October 8, 2026. arena.ai/...ai-alignment-index
- ^1 ^2Arena Team. "Introducing Max in Agent Arena." Arena blog, published October 12, 2026, last updated October 14, 2026. arena.ai/...max-router-tops-agent-arena
- ^1 ^2AlphaSignal Newsroom. "Arena.ai Crowns Claude Opus 5.5 After Rewriting Its Agent Shell Reliability Test." AlphaSignal, October 13, 2026. alphasignal.ai/...its-agent-shell-reliability-test
- ^1 ^2Capital & Compute. "Agent Arena: The AI Agent Leaderboard Explained." Published July 3, 2026, updated October 18, 2026. capitalandcompute.net/...ena-leaderboard-explained
- ^Johnson, Olivia. "Arena $200 Million Raise Puts Its AI Leaderboard Under a Harder Test." Remio, October 9, 2026. remio.ai/...its-ai-leaderboard-under-a-harder-test
- ^Arena Team. "LMArena is now Arena." Arena blog, January 28, 2026. arena.ai/...lmarena-is-now-arena
- ^Singh, S., et al. "The Leaderboard Illusion." arXiv:2504.20879, 2025. arxiv.org/...2504.20879
- ^Bonatti, R., et al. "Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale." arXiv:2409.08264, 2024. arxiv.org/...2409.08264
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation. Every suggestion is reviewed for sourcing before it goes live.
2 revisions · v2 · 3,882 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: xg20 independent verification 2026-10-19; defects fixed
Cite this page: AI Wiki. "Agent Arena." aiwiki.ai, updated 18 Oct 2026, fact-checked 18 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/agent_arena