# RoboArena

> Source: https://aiwiki.ai/wiki/roboarena
> Updated: 2026-11-01
> Fact-checked: 2026-11-01
> Categories: AI Benchmarks, Embodied AI, Model Evaluation, Robotics
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "RoboArena." aiwiki.ai, 1 Nov 2026. https://aiwiki.ai/wiki/roboarena
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**RoboArena** is a distributed, real-world benchmark for generalist robot policies in which volunteer evaluators at different laboratories run double-blind, pairwise (A/B) comparisons of policies on tasks and scenes they choose themselves, and a central server aggregates the preferences into a ranking. It was introduced in the paper "RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies" by Pranav Atreya, Karl Pertsch, Tony Lee and 29 co-authors, posted to arXiv on 22 June 2025 and published in the proceedings of the 9th Conference on Robot Learning (CoRL 2025).[1][2] The benchmark currently runs on the DROID robot platform, a Franka Panda arm setup, and its public leaderboard ranks [vision-language-action models](https://aiwiki.ai/wiki/vision_language_action_model) and other manipulation policies that can be served to that hardware.[1][3]

The design borrows the idea behind crowd-sourced language-model leaderboards such as [Chatbot Arena](https://aiwiki.ai/wiki/lmsys_chatbot_arena): instead of standardizing a fixed task suite, it relies on many independent pairwise judgments collected in diverse conditions.[1] In the original paper, the authors ran more than 600 pairwise real-robot evaluations of seven generalist policies across evaluators at seven academic institutions and reported that the resulting ranking matched an exhaustive "oracle" ranking more closely than a conventional centralized evaluation.[1] As of 1 November 2026, the public leaderboard lists 24 policies, and its "Official" view, which only includes policies with at least 100 A/B evaluations, ranks NVIDIA's DreamZero first and [Physical Intelligence](https://aiwiki.ai/wiki/physical_intelligence)'s pi05_droid second.[4]

## Background

Real-world robot benchmarks have usually tried to make results comparable through standardization: detailed manuals for reproducing scenes, remotely accessible central test stations, or in-person challenges on a fixed date.[1] The RoboArena authors argue that these approaches restrict evaluations to a narrow set of tasks and environments, which suits single-task policies but not generalist policies meant to work across many scenes and instructions. Reproducing lighting, camera angles and object layouts across institutions is also hard, and centralized fleets of evaluation robots are expensive to maintain.[1] Simulation offers reproducibility, but the paper notes that simulators remain imperfect replicas of the real world and sometimes fail to reflect real-world performance.[1]

The paper cites Chatbot Arena and GenAI Arena as inspiration, and distinguishes RoboArena from an earlier cross-institution robotics benchmark by Dasari et al., which required policies to be retrained for every new institution and task.[1] RoboArena relies instead on the fact that modern generalist policies trained on the DROID dataset can be run in new scenes out of the box, so the same policy can be evaluated anywhere a DROID station exists.[1]

## Evaluation protocol

An evaluation session follows a fixed sequence.[1]

1. The evaluator requests a pair of policies from the central server, which samples two policies at random. Evaluators only receive the addresses of remotely hosted policy servers, so they do not know which policies they are testing.
2. The evaluator arranges a scene of their choice, for example by moving the robot to a new location or rearranging objects, and writes a natural-language task instruction.
3. Both policies run back to back on that task until it is completed or a timeout is reached. The evaluator must closely match the initial conditions for the two runs within a pair, but may change scenes freely between pairs.
4. The evaluator reports three kinds of feedback: a continuous progress score from 0 (no progress) to 100 (task completed), a pairwise preference for one policy, and a free-form written explanation of the preference.

The instruction, feedback and full recordings of observations and actions are uploaded to a central database. Apart from the requirement to keep one pair's conditions identical, sessions run asynchronously at any time or place.[1] The authors list four benefits of this decentralized design: it is open-ended because tasks are not fixed, robust because no single party can easily sway double-blind distributed results, scalable because many institutions can contribute, and adaptable because evaluators can raise task difficulty as policies improve.[1] The public leaderboard also records ties, and its A/B evaluation viewer shows each session's instruction, preference and written feedback.[5]

## Ranking model

RoboArena's ranking starts from the Bradley-Terry model, which models the probability that one policy is preferred over another as a sigmoid of the difference between their latent "log-abilities." Such parameters can be fit online with an [Elo-style](https://aiwiki.ai/wiki/elo_rating_system_ai) update or offline by maximum likelihood, and the offline method can be extended to handle ties.[1] The authors argue that standard Bradley-Terry fitting assumes every comparison happens under identical conditions, which is not true when each A/B trial uses a different task: a task that is very easy or very hard for both policies carries little signal, and a specialist policy can beat a stronger generalist on its own subset of tasks.[1]

The paper therefore adds latent "task buckets" to the model. Each bucket has a base difficulty and a prior probability, and each policy has an offset per bucket that lets two policies rank differently on different kinds of tasks. These task parameters are learned only from preference data; the true task identities are never required.[1] Ties are handled with the Davidson extension of Bradley-Terry. The parameters are fitted with an expectation-maximization procedure that alternates between computing each trial's bucket responsibilities and applying clipped Newton updates, re-centering the ability parameters to zero mean after each step.[1] A variant can also incorporate the progress scores.[1] The live leaderboard's transparency tools describe its ranking path as a "Bradley-Terry Davidson" fit.[3]

Because the scores are relative, absolute values on the leaderboard can move when the pool changes. In RoboArena's weekly snapshot of 24 August 2026, every policy carried over from the previous week rose by 244 to 278 points (for example, pi05_droid from 1,599 to 1,843) while a newly added policy entered at -3,884; in the next snapshot those scores were back near their earlier levels.[6] Scores from different dates are therefore not directly comparable.

## The DROID-RoboArena system

The first instantiation of RoboArena uses the DROID platform, which consists of a Franka Panda 7-degree-of-freedom arm, a Robotiq 2F-85 parallel-jaw gripper, a ZED-mini stereo wrist camera and one or more external ZED 2 stereo cameras, mounted on a height-adjustable mobile table.[1] The authors chose it because the arm is dexterous enough for a wide range of manipulation tasks, the mobile table makes it easy to change scenes and viewpoints, the platform comes with the open DROID dataset for training generalist policies, and DROID stations were already deployed at multiple institutions.[1] The paper says the approach could extend to other robots and to cross-embodiment policies, but its experiments cover DROID only.[1]

The system has four components:[1]

| Component | Role |
| --- | --- |
| Policy inference servers | Each submitted policy runs on a remote server hosted by its submitter, so evaluators need no local inference compute and proprietary models can be tested without sharing weights. The paper found the added network latency negligible for the static manipulation tasks typically used on DROID. |
| Evaluation clients | A client script at each robot guides the evaluator through the protocol and talks to the central server and the policy servers. |
| Evaluation database | Stores instructions, scores, preferences, written feedback and rollout videos. |
| Central evaluation server | Assigns policy pairs to evaluators, tracks newly registered or retired policies, and cancels sessions that time out. |

The project's [GitHub](https://aiwiki.ai/wiki/github) repository, created on 14 June 2025 and released under the MIT License, contains code for serving a policy as a remote server, instructions for testing a policy in a simulated DROID environment before submission, and pointers to DROID training code in Physical Intelligence's [openpi](https://aiwiki.ai/wiki/openpi) repository.[7] A policy server declares which observations it needs (for example, image resolution, whether it uses the wrist camera and how many external cameras) and its action space, so the client sends only the required data.[7]

Before a new policy enters the general pool, the organizers check that its server follows the expected input and output formats and then run it in a test environment with a trained evaluator who can intervene if it acts unsafely, which the paper compares to a test driver for autonomous vehicles.[1] To balance supply and demand, the paper describes an "evaluation credit" system in which each pairwise evaluation an evaluator runs earns a credit that can be spent on comparisons involving the evaluator's own policies, and several institutions agreed to sponsor a base budget of evaluations for researchers without robots.[1] The project website describes this as a weekly evaluation budget that can be increased by contributing evaluations.[3]

## Results in the original paper

The paper's policy pool consisted of all publicly available generalist DROID policies that, to the authors' knowledge, worked out of the box in new environments at the time: seven policies built on [PaliGemma](https://aiwiki.ai/wiki/paligemma) or [pi0](https://aiwiki.ai/wiki/pi0) base models, including pi0 with flow matching, pi0-FAST, and five PaliGemma models fine-tuned on DROID with different action representations (flow matching, FAST, FAST+, finite scalar quantization and simple binning).[1] All were run from the public openpi implementation.[1]

To obtain a ground-truth reference, the authors asked evaluators, after each A/B comparison, to also score the remaining five policies on the same task. This produced 4,284 evaluation episodes and an "oracle" ranking based on average progress scores. They compared RoboArena against the conventional DROID evaluation used in earlier work, which tests every policy on a fixed set of 17 tasks with 44 episodes per policy.[1] The main findings were:

- RoboArena rankings, built from 612 pairwise comparisons across seven universities, correlated more closely with the oracle ranking than the conventional evaluation did, measured by Pearson correlation and mean maximum rank violation (MMRV). The task-aware model gave more accurate rankings than plain Elo or standard Bradley-Terry.[1]
- RoboArena reached high-quality rankings within about 100 pairwise comparisons, matching the convergence speed of conventional evaluation, and kept improving with more data.[1]
- In a simulated scenario where tasks became harder over time and weaker policies were replaced by stronger ones, RoboArena still scored a Pearson correlation of 0.838 and an MMRV of 0.058 against the oracle, compared with 0.692 and 0.141 for the conventional single-lab evaluation.[1]
- Averaging progress scores alone also produced good rankings, but in 11% of A/B evaluations the preference disagreed with the progress scores. Evaluators regularly gave both policies the same progress score while still preferring one, for example because it acted more swiftly or confidently. The authors recommend reporting both kinds of ranking.[1]
- Both the oracle and RoboArena rankings found that expressive action representations beat simple binning tokenization and that discrete tokenization (FAST, FSQ) beat diffusion-style policies in these language-conditioned tests. The strongest policy in the pool was pi0-FAST-DROID.[1]

The paper also prototyped LLM-assisted analysis. A vision-language model categorized each task and described scene conditions such as lighting and clutter, and a language model wrote a report on each policy that cites specific evaluation videos as evidence; the paper names OpenAI's GPT-4.5 for categorization and o3 for the reports.[1] Task categories matched a human expert's labels about 95% of the time on 448 samples, although the paper's accuracy table lists the model for that result as GPT-4o.[1] Across all policies, the authors found that direct object manipulation such as pick-and-place, pushing and simple opening or closing was solved more reliably than tool use, cloth manipulation, precise alignment or multi-step instructions.[1]

## Launch, venue and CoRL 2025 challenge

The arXiv paper appeared on 22 June 2025, with a revised version on 29 November 2025.[1] It was published in the Proceedings of the 9th Conference on Robot Learning, PMLR volume 305, pages 336-364.[2] The authors are affiliated with UC Berkeley, [Stanford University](https://aiwiki.ai/wiki/stanford_university), the University of Washington, the Université de Montréal, NVIDIA, the University of Pennsylvania, UT Austin and Yonsei University; Pertsch, Atreya and Lee are listed as core contributors, and the advisers include [Sergey Levine](https://aiwiki.ai/wiki/sergey_levine), [Chelsea Finn](https://aiwiki.ai/wiki/chelsea_finn) and [Percy Liang](https://aiwiki.ai/wiki/percy_liang).[1][3]

Alongside the paper, the team ran a RoboArena challenge as part of a CoRL 2025 workshop on generalist policies in the wild, held on 27 September 2025.[8] Challenge materials and simulated environments were released on 15 July 2025, final policies were due on 13 September 2025, and RoboArena evaluations from 13 to 20 September decided the final ranking. Participants received extra evaluation credits, and the organizers encouraged entries from machine-learning researchers without access to a robot.[8][9] The project website says the benchmark "will be running live through Dec 2026, with potential extensions beyond."[3]

## Public leaderboard

The leaderboard at robo-arena.github.io has two views. "Official" includes only policies with at least 100 A/B evaluations, for more stable rankings; "All Policies" also shows newer entries with fewer evaluations, flagged as low sample.[4] Each row gives the score, its standard deviation, the number of A/B evaluations and whether the policy is open source, and a status column for each policy server.[4] The site maps some internal policy identifiers to display names, for example showing "dreaming_zebra" as DreamZero and "j2-vla" as Spirit v1.6.[3]

The Official view, last updated on 1 November 2026, listed nine policies:[4]

| Rank | Policy | Score | SD | A/B evaluations | Open source |
| --- | --- | --- | --- | --- | --- |
| 1 | DreamZero | 1,735 | 42.6 | 190 | Yes |
| 2 | pi05_droid | 1,608 | 30.7 | 745 | No |
| 3 | pi0_fast_droid | 1,582 | 29.6 | 941 | No |
| 4 | paligemma_vq_droid | 1,546 | 29.9 | 952 | Yes |
| 5 | paligemma_diffusion_droid | 1,533 | 29.7 | 949 | Yes |
| 6 | paligemma_fast_specialist_droid | 1,531 | 29.8 | 1,057 | Yes |
| 7 | paligemma_fast_droid | 1,516 | 29.9 | 1,062 | Yes |
| 8 | pi0_droid | 1,461 | 29.8 | 1,120 | No |
| 9 | paligemma_binning_droid | 786 | 50 | 629 | Yes |

In the All Policies view on the same date, Spirit v1.6 ranked first with 1,788 points from 25 evaluations (standard deviation 106), ahead of DreamZero, followed by low-sample entries named Apricot (1,661 from 63 evaluations) and ReDROID-0.0 (1,634 from 6 evaluations).[4] DreamZero is a [world action model](https://aiwiki.ai/wiki/world_action_model) from NVIDIA researchers, whose February 2026 paper says they open-sourced code to run the RoboArena benchmark, and pi05_droid is the DROID version of Physical Intelligence's [pi0.5](https://aiwiki.ai/wiki/pi_0_5).[10][4]

Several outside reports have cited earlier readings of the board. An NVIDIA Technical Blog post by Moritz Reuss on 15 June 2026 reproduced an April 2026 snapshot in which DreamZero scored 1,750, pi0.5 1,622, pi0-FAST 1,592 and pi0 1,475, and cautioned that this was "not proof that WAMs are the better default."[11] On 5 June 2026, The Next Web reported an announcement by [Spirit AI](https://aiwiki.ai/wiki/spirit_ai) that its Spirit v1.6 model had scored 1,924, ahead of NVIDIA's Cosmos3-Nano-Policy at 1,881 and DreamZero at 1,763, calling it the first time a Chinese model had led RoboArena.[12] NVIDIA's own launch release for [Cosmos 3](https://aiwiki.ai/wiki/nvidia_cosmos_3), dated 31 May 2026, said that among open models it ranked first on RoboLab and RoboArena "for action policy."[13] Because the scores are relative and change as policies are added, these figures describe the board at those moments rather than current standings.[6]

## Evaluation data and transparency

RoboArena publishes its evaluations. The A/B evaluation viewer and its summary data, generated on 1 November 2026, count 3,994 public A/B evaluations covering 24 policies and 120 policy pairs, with a 14.4% tie rate. The first public evaluation is dated 15 April 2025 and the most recent 12 September 2026.[5][14] Evaluations carry a free-text organization label; the summary counts 28 such labels, but ten of them are variant spellings of Berkeley and a few are not institutions (for example "CS224R Project").[14] The largest single contributor is the label "frodobots," with 2,301 evaluations (about 58% of the total) since August 2025, followed by Berkeley (497), UT Austin (390), NVIDIA (219), Stanford (133) and UPenn (116).[14] The viewer reports how the official ranking would change if any one organization's evaluations were removed, using the same ranking algorithm.[3]

A data snapshot on [Hugging Face](https://aiwiki.ai/wiki/hugging_face), "RoboArena Dataset Snapshot 2026-07-17," contains the rollouts, task-success scores, preference annotations and metadata collected from the benchmark's start through 17 July 2026: 3,883 evaluation sessions, 10,783 policy episodes and 27,148 MP4 videos, released under the MIT License.[15]

## Related work

RoboArena's authors and other groups have worked on simulated evaluations meant to approximate it more cheaply.

- **PolaRiS** (December 2025), by Arhan Jain, Karl Pertsch and colleagues including several RoboArena authors, turns short video scans of real scenes into interactive simulation environments through neural reconstruction. On DROID, the authors report an average Pearson correlation of 0.9 between PolaRiS and real-world performance, and a correlation of 0.98 with the average progress scores reported in RoboArena. They add that their simulated evaluations capture only a small subset of what RoboArena tests, leaving out non-rigid and soft-body interactions and complex contact behavior.[16]
- **RobotArena ∞** (October 2025; published at ICLR 2026), from a separate team whose authors include Katerina Fragkiadaki and Yonatan Bisk, automatically converts video demonstrations from robot datasets into simulated digital twins, then scores VLA policies with vision-language-model-guided evaluation and crowd-sourced human preference comparisons.[17][18] Its paper cites RoboArena as related work; it is a separate project rather than a RoboArena extension.[17]

## Limitations

The paper lists four main limitations.[1] Its experiments cover only the DROID platform, and extending the method to multiple embodiments, including policies that can only run on specific robots, is left to future work. Because tasks and scenes are unconstrained, RoboArena cannot easily run controlled experiments that vary one factor at a time, so it complements targeted generalization studies. The authors have not tested its robustness against deliberately adversarial evaluators who give random ratings or misleading feedback. Finally, they note that crowd-sourced evaluations, like any metric, are subject to over-optimization under [Goodhart's law](https://aiwiki.ai/wiki/goodharts_law).[1]

The public data show further practical constraints. Low-sample entries carry large uncertainties (several have fewer than 10 evaluations), more than half of all public evaluations come from one organization label, and, as of 1 November 2026, the most recent public evaluation was dated 12 September 2026.[4][14] The leaderboard only covers policies that can be served to a DROID Franka setup.[1][3]

## See also

- [Vision-language-action model](https://aiwiki.ai/wiki/vision_language_action_model)
- [Robot foundation model](https://aiwiki.ai/wiki/robot_foundation_model)
- [LMArena](https://aiwiki.ai/wiki/lmarena_org)
- [RoboDojo](https://aiwiki.ai/wiki/robodojo)
- [Physical Intelligence](https://aiwiki.ai/wiki/physical_intelligence)
- [Open X-Embodiment](https://aiwiki.ai/wiki/open_x_embodiment)
- [Physical AI](https://aiwiki.ai/wiki/physical_ai)

## References

1. Atreya, Pranav; Pertsch, Karl; Lee, Tony; et al. "RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies." arXiv:2506.18123, v1 22 June 2025, v2 29 November 2025. https://arxiv.org/abs/2506.18123
2. Atreya, Pranav; et al. "RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies." Proceedings of The 9th Conference on Robot Learning, PMLR 305:336-364, 2025. https://proceedings.mlr.press/v305/atreya25a.html
3. RoboArena. Project website. Accessed 1 November 2026. https://robo-arena.github.io/
4. RoboArena. "Policy Leaderboard" (Official and All Policies views). Last updated 1 November 2026, accessed 1 November 2026. https://robo-arena.github.io/leaderboard
5. RoboArena. "A/B Evaluation Viewer." Accessed 1 November 2026. https://robo-arena.github.io/results
6. RoboArena. Weekly leaderboard snapshots (leaderboard_history data feed used by the RoboArena website), June to October 2026. Accessed 1 November 2026. https://roboarena-api-domain-name.online/api/leaderboard_history
7. robo-arena/roboarena. GitHub repository, MIT License. Accessed 1 November 2026. https://github.com/robo-arena/roboarena
8. "Workshop on Generalist Policies in the Wild & Robo-Arena Challenge." Conference on Robot Learning 2025 workshop site. https://sites.google.com/view/corl-roboarena
9. RoboArena. "RoboArena x CoRL 2025 Challenge." https://robo-arena.github.io/challenge
10. Ye, Seonghyeon; et al. "World Action Models are Zero-shot Policies." arXiv:2602.15922, 17 February 2026. https://arxiv.org/abs/2602.15922
11. Reuss, Moritz. "Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models." NVIDIA Technical Blog, 15 June 2026. https://developer.nvidia.com/blog/pretrained-to-imagine-fine-tuned-to-act-the-rise-of-world-action-models/
12. Popa, Darius. "A Chinese startup just dethroned Nvidia on the benchmark Nvidia helped build." The Next Web, 5 June 2026. https://thenextweb.com/news/spirit-ai-beats-nvidia-roboarena-physical-ai
13. NVIDIA Newsroom. "NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI." 31 May 2026. https://nvidianews.nvidia.com/news/nvidia-launches-cosmos-3-the-open-frontier-foundation-model-for-physical-ai
14. RoboArena. Evaluator organization statistics (transparency_summary data feed used by the A/B Evaluation Viewer), generated 1 November 2026. https://roboarena-api-domain-name.online/api/transparency_summary
15. RoboArena. "RoboArena Dataset Snapshot 2026-07-17." Hugging Face dataset, 17 July 2026. https://huggingface.co/datasets/RoboArena/DataDump_07-17-2026
16. Jain, Arhan; Zhang, Mingtong; Arora, Kanav; et al. "PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies." arXiv:2512.16881, 18 December 2025. https://arxiv.org/abs/2512.16881
17. Jangir, Yash; Zhang, Yidi; Lo, Pang-Chi; et al. "RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation." arXiv:2510.23571, 27 October 2025. https://arxiv.org/abs/2510.23571
18. ML Anthology. "RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation." ICLR 2026. https://mlanthology.org/iclr/2026/jangir2026iclr-robotarena/

