Chatbot Arena

Chatbot Arena (now branded simply as Arena, and previously known as LMArena) is a crowdsourced evaluation platform for large language models that ranks AI systems based on human preferences through anonymous, blind, side-by-side comparisons.^[1] Users type any prompt, two unnamed models answer, the user votes for the better response, and the votes are aggregated into a single numerical rating per model. Created by researchers at UC Berkeley in May 2023, it grew from a small academic project into the most widely cited public benchmark for comparing frontier AI models.^[3] By early 2026 the platform drew more than 5 million monthly users across 150 countries generating over 60 million conversations per month and had spun out as an independent company valued at $1.7 billion; in October 2026 it raised a $200 million Series B at a $3.1 billion valuation and reported 62 million votes across its text, vision, code, search, video and image arenas.^[8][11][12]

The founding paper defines it as "an open platform for evaluating LLMs based on human preferences" whose methodology "employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing."^[1]

Origins and founding

Chatbot Arena emerged from UC Berkeley's Sky Computing Lab in the spring of 2023. The project was led by a group of graduate students and faculty, including Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Anastasios Nikolas Angelopoulos, Hao Zhang, and professors Joseph E. Gonzalez and Ion Stoica.^[1] The team collectively operated under the name LMSYS Org (Large Model Systems Organization), which also maintained related open-source projects like Vicuna and FastChat.

The original motivation was straightforward: existing benchmarks for language models were too narrow. Benchmarks like MMLU or HellaSwag tested specific capabilities under controlled conditions, but they did not capture how well a model actually performed in open-ended conversation with real users.^[3] Zheng and his collaborators wanted a system that would let ordinary people compare models head-to-head on whatever questions they cared about, with the results aggregated into a global ranking.^[1]

The first version of Chatbot Arena went live in May 2023.^[3] Within months, the platform had deployed over 45 models and collected more than 130,000 valid votes.^[10] In March 2024, the research paper describing the platform (Chiang, Zheng et al., 2024), later published at ICML 2024, reported about 240,000 votes from more than 90,000 users collected since April 2023, and the leaderboard was being cited in press releases by OpenAI, Google, Anthropic, and other major AI labs as evidence of their models' performance.^[1]

How does Chatbot Arena work?

The core mechanic of Chatbot Arena is a blind pairwise comparison. When a user visits the platform, they can type any prompt they want. The system sends the prompt to two randomly selected models and displays both responses side by side. The user does not know which models they are evaluating. After reading both responses, the user votes for whichever answer they prefer, or they can declare a tie. Only after voting are the model identities revealed.^[3]

This design addresses several problems with traditional benchmarks. Because users write their own prompts, the evaluation covers a vast and continually refreshed distribution of real-world use cases, from creative writing and coding to math problems and factual questions. Because the comparison is blind, there is no brand-name bias influencing votes. And because the voting is binary (A vs. B), users do not need to calibrate an absolute quality scale.^[1]

How a battle works step by step

The battle process follows a carefully designed workflow:

1. Prompt entry. The user types any prompt into a single input box. There are no preset prompts or restricted topics; users are free to ask anything.
2. Model sampling. The system selects two models from the pool. The sampling is not uniform: models with similar ratings and higher uncertainty are paired more frequently to maximize the information gained from each vote.^[1] This non-uniform sampling concentrates votes where they are most needed.
3. Response generation. Both models receive the identical prompt and generate responses simultaneously. The responses are displayed side by side, labeled only as "Model A" and "Model B."
4. Position randomization. The left-right assignment of models is randomized to mitigate position bias (the tendency for users to prefer whichever response appears on a particular side).^[1]
5. Voting. The user selects one of four options: "A is better," "B is better," "Tie," or "Both are bad." The user can also continue the conversation with multi-turn exchanges before voting.^[3]
6. Reveal. After voting, the model identities are revealed, and the user can see which models they compared.

Crowdsourcing quality control

With millions of votes from anonymous users, quality control is essential. The Arena employs several strategies:

Anomalous-user detection. The 2024 paper describes a first-step statistical test that flags anomalous users, such as those whose inputs are highly repetitive or meaningless, by testing whether their votes are statistically unlike those of the existing user base.^[1]

Demographic awareness. The platform acknowledges that its user base skews toward English-speaking, technically sophisticated users. This is a known limitation rather than a controlled variable; the ratings reflect this population's preferences, which may not generalize to other demographics.^[1]

Elo and Bradley-Terry ratings

The platform originally used the Elo rating system, borrowed from competitive chess, to convert pairwise vote outcomes into a single numerical score for each model.^[3] In the Elo system, when a lower-rated model beats a higher-rated one, the ratings shift more dramatically than when the expected winner prevails.

The Bradley-Terry model

LMSYS later transitioned to the Bradley-Terry (BT) model, a statistical framework from 1952 that estimates the probability of one player beating another based on their latent "strength" parameters.^[1] The BT model works by assigning each model i a strength parameter p_i, and the probability that model i beats model j is calculated as:

P(i beats j) = p_i / (p_i + p_j)

The strength parameters are estimated via maximum likelihood estimation (MLE) across all collected matchups. This approach is mathematically related to Elo (Elo ratings can be viewed as an online approximation of BT coefficients), but fitting BT via MLE across all data produces more stable and statistically principled rankings.^[1]

The key difference is that the Elo system weights recent games more heavily (it is an online algorithm that updates after each game), while the BT model treats all games equally and recomputes from scratch. For a platform where most models are static (unlike chess players who improve or decline), the BT model is more appropriate, although hosted proprietary models can change without notice.^[10]

The platform computes 95% confidence intervals through bootstrapping: the rating calculation is repeated many times with random subsamples of the vote data, and the spread of results indicates how reliable each model's ranking is. Models with fewer votes have wider confidence intervals.^[1]

Category-specific leaderboards

The Arena also maintains separate category-specific leaderboards, including rankings for:

This lets users see which models excel in particular domains rather than relying solely on the overall ranking.

Style control

A significant methodological innovation introduced in August 2024 was style control, which attempts to separate what a model says from how it says it.^[5] Research by the LMSYS team found that user preferences were significantly influenced by stylistic factors, particularly response length. As the team put it, "when analyzing the style coefficients, we found that length was the dominant style factor."^[5]

Style factors and their impact

The team identified four primary style features and their influence coefficients:

Response length emerged as the overwhelmingly dominant style factor, with all other markdown formatting effects being second-order.^[5] Models that generated longer, more formatted responses tended to win more battles, not necessarily because their content was better, but because users preferred the appearance of thoroughness.^[5]

Methodology

The style control approach uses a modified Bradley-Terry regression model that treats style as an independent variable. Each style feature is measured as a normalized difference between the two responses:

style_difference = normalize((feature_A - feature_B) / (feature_A + feature_B))

This proportional normalization accounts for context: a 20-token difference matters more when comparing 20-token versus 40-token responses than when comparing 500-token versus 520-token responses.^[5]

Impact on rankings

When style was controlled for on the overall leaderboard, notable ranking shifts occurred:

Models that rose (substance over style):

• Claude 3.5 Sonnet: climbed from rank 6 to rank 4
• Claude 3 Opus: climbed substantially

Models that fell (style over substance):

• GPT-4o-mini: dropped from rank 6 to rank 11
• Grok-2-mini: dropped from rank 6 to rank 18

Llama-3.1-405B held its position at rank 6 with no change.^[5] These shifts revealed that some models were achieving high Arena scores partly through verbose, well-formatted responses rather than superior reasoning or factual accuracy.^[5]

Arena-Hard-Auto

Recognizing that live human evaluation is slow and expensive, the LMSYS team developed Arena-Hard-Auto as a complementary offline benchmark.^[4] Arena-Hard-Auto consists of 500 challenging prompts sampled from real Chatbot Arena conversations, specifically chosen from 250 high-quality topic clusters (two prompts per cluster) where the questions were difficult enough to reliably separate strong models from weaker ones.^[4]

The evaluation uses an LLM-as-a-judge approach: a powerful model (originally GPT-4-1106-preview) scores each test model's responses against a baseline model.^[4] To mitigate position bias, every comparison is run twice with the response order swapped, yielding 1,000 judgments per model. The results are then converted to Bradley-Terry scores.^[4]

LMSYS reported that Arena-Hard-Auto v0.1 had higher agreement with Chatbot Arena rankings (89.1%) than MT-Bench or AlpacaEval 2.0 LC, making it a useful proxy when live testing is impractical.^[4] The prompts and evaluation code are publicly available on GitHub.

Leaderboard snapshot (March 2026)

The Arena leaderboard is updated continuously as new votes come in. The following table shows the top 15 models on the overall text leaderboard as of March 5, 2026, when it held 5,430,034 votes across 323 models.^[9]

Ratings shift as new votes accumulate. Models marked as "Preliminary" have fewer votes and wider confidence intervals.^[9]

The Leaderboard Illusion controversy

In April 2025, a research paper titled "The Leaderboard Illusion" (authored by Shivalika Singh and colleagues from Cohere Labs, the Allen Institute for AI, Princeton, Stanford, MIT, the University of Waterloo, and the University of Washington) accused LMArena of enabling systematic manipulation of the leaderboard by major AI companies.^[6] The paper's central claim was that LMArena had allowed companies like Meta, OpenAI, Google, and Amazon to privately test multiple model variants on the platform, then publicly reveal only the scores of the best-performing variants while retracting or burying the rest.^[6]

The paper also documented uneven access to Arena data: it estimated that Google and OpenAI received roughly 19.2% and 20.4% of all data on the platform respectively, while a combined 83 open-weight models received only about 29.7% of the total.^[6]

The Llama 4 scandal

The most high-profile case involved Meta's release of Llama 4 in early 2025. According to the paper, Meta had privately tested 27 different model variants on Chatbot Arena in the lead-up to the Llama 4 release.^[6] Only one variant, which had been specifically optimized for human preference (a quality that tends to perform well in Arena's preference-based voting), was publicly listed near the top of the leaderboard. Meta never released this optimized model to the public; the vanilla Llama 4 version that users could actually download performed significantly worse.^[7]

The specific model submitted was "Llama-4-Maverick-03-26-Experimental," a custom variant of Llama 4 Maverick that had been tuned for human preference. It ranked second on the overall leaderboard, just behind Google's Gemini 2.5 Pro, with the Elo score of 1417 that Meta highlighted in its press release.^[14] Researchers observed stark differences in behavior between the Arena version and the downloadable model.^[15] LMArena then released more than 2,000 head-to-head battle results for public review and said early analysis showed that "style and model response tone was an important factor."^[16] Given the documented length bias in Arena voting, such stylistic tuning can inflate a model's ranking.^[5] When the unmodified Maverick was later listed, it ranked below models including OpenAI's GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro.^[17]

LMArena said that "Meta's interpretation of our policy did not match what we expect from model providers" and that Meta "should have made it clearer" that the Arena entry "was a customized model to optimize for human preference," adding that it was updating its leaderboard policies.^[16] Meta's VP of Generative AI, Ahmad Al-Dahle, denied a rumor that Meta had trained on test sets ("that's simply not true and we would never do that"), attributing the variable quality people were seeing to "needing to stabilize implementations."^[14]

This is a textbook example of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. By optimizing specifically for Arena performance rather than general capability, companies could inflate their rankings without delivering corresponding improvements to users.

Broader manipulation findings

The Leaderboard Illusion paper documented similar patterns with other companies:

• Multiple companies submitted "experimental" model variants that differed from their public releases.
• Some companies tested dozens of variants privately before announcing the highest-scoring one.
• Private testing and unequal data access together led to overfitting to Arena-specific dynamics rather than general model quality.^[6]

LMArena co-founder Ion Stoica pushed back against the paper, telling TechCrunch it was full of "inaccuracies" and "questionable analysis."^[7] LMArena said it had published information on pre-release testing since March 2024 and rejected the paper's call to publish scores for unreleased models, but indicated it would create a new sampling algorithm.^[7]

The controversy highlighted a broader tension in AI evaluation: the more influential a benchmark becomes, the greater the incentive to game it.

Transition from research project to company

Chatbot Arena's trajectory from academic side project to venture-backed startup unfolded rapidly. The company's founders, funding rounds, and product lines are covered in more detail in the LMArena article.

The founding team shifted into corporate roles: Anastasios Angelopoulos became CEO, Wei-Lin Chiang became CTO, and Ion Stoica remained involved as a co-founder.^[13] Investors across the two rounds included Andreessen Horowitz, UC Investments, Felicis, Lightspeed Venture Partners, Kleiner Perkins, Laude Ventures, and The House Fund.^[11]

The rebranding to "Arena" in January 2026 reflected the platform's expansion beyond text-only language models. By that point, Arena had added support for image generation models, video generation, and multimodal systems, making the "LM" (language model) prefix too restrictive.^[8] The company framed the change in its announcement: "Now that our platform has become the de facto Arena, we have outgrown the 'LM,' and it's time to drop it."^[8]

Statistical methodology details

Bootstrapping for confidence intervals

The Arena computes confidence intervals by repeatedly resampling the vote data with replacement (bootstrapping). For each bootstrap iteration, a new set of Bradley-Terry coefficients is estimated, producing a distribution of possible ratings for each model. The 2.5th and 97.5th percentiles of this distribution form the 95% confidence interval.^[1]

Models with fewer votes have wider confidence intervals. On the March 5, 2026 text leaderboard, for example, gemini-3-pro had 39,673 votes and a margin of plus or minus 4 points, while gpt-5.4-high had 2,290 votes and a margin of plus or minus 12.^[9] The platform marks models with wide intervals as "Preliminary" to signal that their rankings are less reliable.^[9]

Vote weighting and sampling

Two further choices shape how votes become rankings:

Inverse-probability weighting. Because model pairs are sampled non-uniformly, the Bradley-Terry fit weights each vote by the inverse of its pair's sampling probability, so that the scores target a uniform distribution over model pairs.^[1]

Version pinning. Rather than decaying old votes, LMSYS tries to pin specific API versions of hosted models and has split versions such as GPT-4-0314 and GPT-4-0613 into separate entries, since a provider can change a hosted model without notice.^[10]

Why does Chatbot Arena matter?

Chatbot Arena's influence on the AI industry is hard to overstate. Its authors describe it as "one of the most referenced LLM leaderboards, widely cited by leading LLM developers and companies," partly because it reflects real user preferences at scale rather than narrow academic tasks.^[1]

Several factors contributed to this influence. First, the evaluation is grounded in human judgment rather than automated metrics, which makes it resistant to the kind of benchmark-specific optimization that plagues other evaluations (though, as the Llama 4 scandal showed, not entirely immune). Second, the prompts come from real users rather than a fixed test set, which means the benchmark continuously evolves with user needs. Third, the pairwise comparison design is intuitive: anyone can understand "Model A beat Model B" without needing to interpret abstract accuracy scores.^[1]

At the same time, the platform has limitations. The user population skews toward English-speaking, technically sophisticated users, which means the rankings may not reflect performance for other demographics.^[1] The preference signal is noisy; users vote based on subjective criteria that vary from person to person. And as the Leaderboard Illusion controversy showed, the system is not immune to strategic manipulation by well-resourced companies.^[6]

Current state

As of early 2026, Arena operates at arena.ai with support for text, image, and video model evaluation.^[8] In January 2026 the company said it had more than 5 million monthly users across 150 countries who generate over 60 million conversations per month.^[8] By October 2026 it reported 62 million votes across text, vision, code, search, video and image, 350 million sessions across the platform, and more than 1,000 new model evaluations.^[12] By December 2025 the company reported an annualized consumption run rate of $30 million, reached less than four months after launching its commercial evaluation service.^[11] That figure reached $100 million by June 2026,^[13] and in October 2026 the company said it had exceeded $100 million in annualized revenue.^[12] Arena-Hard-Auto continues to serve as an offline complement for rapid model comparison.

Since its Series A the company has expanded into agent evaluation: Agent Arena logged 7 million sessions in under five months, and in October 2026 Arena introduced an Alignment Index that scores agents on unauthorized actions, false attribution and deceptive completion.^[12] The same month it raised a $200 million Series B at a $3.1 billion valuation, co-led by Lightspeed Venture Partners and Khosla Ventures.^[12]

Continue reading

• DDynabenchDynabench is an open-source artificial intelligence benchmarking platform that runs in a web browser and supports human-and-model-in-the-loop dataset creationListed in See also
• MModelA model in machine learning is a representation used to map inputs to outputs, assign probabilities, describe relationships, or choose actions.Mentioned 81 times above
• Large Language ModelA large language model (LLM) is an artificial intelligence system built on a transformer neural network with billions to trillions of parameters, trained on massive text corpora to predict the next token in a…Linked in the introduction
• LLMArenaLMArena is a crowdsourced artificial intelligence evaluation platform and company that ranks large language models by having anonymous users vote on which of two blind, side-by-side model responses they preferMentioned 11 times above
• BBenchmark (AI)In artificial intelligence and machine learning, a benchmark is a specified evaluation used to compare systems under common conditions.Mentioned 8 times above
• AArena-HardArena-Hard (and its evaluation tool Arena-Hard-Auto) is an automatic large language model (LLM) benchmark developed by the team behind Chatbot Arena that scores instruction-tuned models on 500 challengingMentioned 5 times above

More to read

• Generative AI
• Claude 3 Opus
• Vicuna (language model)
• Elo rating system (AI model ranking)
• MT-Bench
• LLM-as-a-judge

Topics:AI BenchmarksLarge Language Models

See also

• Dynabench
• AI model comparison finder

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., & Stoica, I. (2024). "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference." *Proceedings of the 41st International Conference on Machine Learning (ICML 2024)*. arxiv.org/...2403.04132
2. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." *Advances in Neural Information Processing Systems 36 (NeurIPS 2023)*. arxiv.org/...2306.05685
3. ^1 ^2 ^3 ^4 ^5 ^6LMSYS Org. (2023). "Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings." LMSYS Blog. lmsys.org/...2023-05-03-arena
4. ^1 ^2 ^3 ^4 ^5LMSYS Org. (2024). "From Live Data to High-Quality Benchmarks: The Arena-Hard Pipeline." LMSYS Blog. lmsys.org/...2024-04-19-arena-hard
5. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8LMSYS Org. (2024). "Does Style Matter? Disentangling Style and Substance in Chatbot Arena." LMSYS Blog. lmsys.org/...2024-08-28-style-control
6. ^1 ^2 ^3 ^4 ^5 ^6Singh, S., Nan, Y., Wang, A., et al. (2025). "The Leaderboard Illusion." arxiv.org/...2504.20879
7. ^1 ^2 ^3Zeff, M. (2025). "Study accuses LM Arena of helping top AI labs game its benchmark." *TechCrunch*. techcrunch.com/...g-top-ai-labs-game-its-benchmark
8. ^1 ^2 ^3 ^4 ^5Arena. (2026). "LMArena is now Arena." Arena Blog. arena.ai/...lmarena-is-now-arena
9. ^1 ^2 ^3 ^4Arena. "Text Arena leaderboard" (snapshot of March 5, 2026). web.archive.org/...text
10. ^1 ^2 ^3LMSYS Org. (2023). "Chatbot Arena: New Models & Elo System Update." LMSYS Blog. lmsys.org/...2023-12-07-leaderboard
11. ^1 ^2 ^3 ^4Bort, J. (2026). "LMArena lands $1.7B valuation four months after launching its product." *TechCrunch*. techcrunch.com/...nths-after-launching-its-product
12. ^1 ^2 ^3 ^4 ^5 ^6Arena. (2026, October 8). "Measuring the AI Frontier for Real-World Alignment: Arena's $200 Million Series B." Arena Blog. arena.ai/...series-b
13. ^1 ^2Temkin, M. (2026, June 29). "Arena, the AI leaderboard everyone uses, is now a $100M business." *TechCrunch*. techcrunch.com/...yone-uses-is-now-a-100m-business
14. ^1 ^2Robison, K. (2025, April 8). "Meta got caught gaming AI benchmarks." *The Verge*. theverge.com/...llama-4-maverick-benchmarks-gaming
15. ^Wiggers, K. (2025, April 7). "Meta exec denies the company artificially boosted Llama 4's benchmark scores." *TechCrunch*. techcrunch.com/...oosted-llama-4s-benchmark-scores
16. ^1 ^2lmarena.ai [@lmarena_ai]. (2025, April 8). Post on X about Llama-4-Maverick-03-26-Experimental. x.com/...1909397817434816562
17. ^Wiggers, K. (2025, April 11). "Meta's vanilla Maverick AI model ranks below rivals on a popular chat benchmark." *TechCrunch*. techcrunch.com/...vals-on-a-popular-chat-benchmark

Show all 17 references