RoboDojo

RoboDojo is a benchmark for generalist robot manipulation policies that combines a simulation suite of 42 tasks with a real-robot suite of 18 tasks, a standardized physical test platform called RoboDojo-RealEval, and a public leaderboard.^[1][2] It was introduced in the arXiv paper "RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies" (arXiv 2607.04434), first submitted on 5 July 2026 by 44 authors led by Tianxing Chen and Yue Chen, with the Multimedia Laboratory (MMLab) at the University of Hong Kong as the lead group.^[1][8] The simulation tasks run in NVIDIA Isaac Sim and are grouped into five capability dimensions (Generalization, Memory, Precision, Long-Horizon and Open), while the real-world tasks run on three bimanual robot arms: ARX X5, Piper and Piper X.^[1] Policies connect to both settings through XPolicyLab, a companion codebase that wraps many existing robot models behind one interface.^[1][6]

The paper's initial results were low across the board. The best policy averaged an 8.80% success rate in simulation and 12.8% in the real world, against 76.03% and 100% for expert human teleoperators under the same rules.^[1][8] Since the July 2026 launch the leaderboard has grown to 66 simulation entries and 13 real-world entries, as of its 30 October 2026 update.^[3] The robotics-evaluation company Robocurve built its RoboDojo-RC Tier 1 benchmark from six adapted RoboDojo real-world tasks.^[11]

Background

RoboDojo is aimed at generalist manipulation policies such as vision-language-action models and other robot foundation models. Its authors argue that many existing benchmarks "rely on simple, short-horizon, or skill-narrow tasks" that often share similar manipulation patterns, and that most evaluations take place either only in simulation or only on real hardware.^[1] Simulation is cheap and repeatable but misses physical deployment problems, while real-robot testing gives direct evidence but is "costly, time-consuming, and difficult to reproduce" because hardware, scene resets, protocols and deployment interfaces differ from lab to lab.^[1]

The paper places RoboDojo next to simulation suites such as CALVIN, LIBERO, RoboTwin, RMBench, GarmentLab and SimplerEnv, and real-world efforts such as ManipulationNet, RoboChallenge and RoboArena.^[1] Chinese tech outlets QbitAI and Jiqizhixin both introduced RoboDojo as the work of the team behind the RoboTwin series of bimanual simulation benchmarks.^[9][10] The paper compares RoboDojo's generalization tasks directly with RoboTwin 2.0, noting that RoboTwin 2.0 uses at most 10 clutter objects while RoboDojo randomizes up to 25.^[1]

Design

Simulation benchmark

The 42 simulation tasks use an ARX X5 bimanual platform with the two arm bases 0.6 m apart.^[1] The simulator is built on Isaac Sim and Isaac Lab, reusing the MagicSim infrastructure for scene construction and object simulation. Each task is defined by YAML files that specify assets, layouts, randomization ranges and success conditions, and scene sampling is controlled by deterministic seeds.^[1] The asset library contains rigid, articulated and deformable objects; some rigid objects were reconstructed from real household items with Meshy AI, and some deformable assets come from ClothesNet.^[1] The paper says the representative tasks cover 24 manipulation skills, including grasping, pushing, stacking, insertion, folding and tool use.^[1]

Source: RoboDojo paper, section 3.1.^[1]

A distinctive part of the simulator is what the authors call heterogeneous parallelism. Standard vectorized simulation clones the same scene template across parallel environments; RoboDojo lets each parallel environment hold a different scene, with different objects, object counts, clutter and articulation structures, while sharing one batched stepping interface.^[1] Episode seeds can also be split across GPUs as separate simulator processes.^[1]

Real-world benchmark

The real-world benchmark has six tasks on each of three collaborative bimanual platforms. The authors say they chose a compact, hard task set because real-robot testing is limited by hardware cost, execution time, human reset effort and safety.^[1] The real tasks are deliberately not one-to-one copies of simulation tasks: the paper says the real benchmark "is not intended to measure direct sim-to-real transfer on matched tasks" and is instead a complementary test of physical deployment.^[1]

Source: RoboDojo paper, Table 2 task order and Appendix J.^[1]

The tasks range from short manipulation (stack bowls, cap a pen) to multi-stage chores. For example, make_food asks the robot to place a cutting board, put a steak and knife on it, put vegetables and shrimp in a pot, move the pot to the stove and cover it; cover_blocks requires the robot to remember which lid hides which colored block.^[1]

Training data

RoboDojo supplies the demonstration data that policies are fine-tuned on. For simulation it provides 3,500 trajectories (1,859,602 frames, 20.66 hours at 25 Hz): 100 per task for the 34 Generalization, Memory, Long-Horizon and Precision tasks, plus 100 domain-randomized "DLC" trajectories of simple pick-and-place to widen visual exposure.^[1] The Open tasks have no training data by design. Simulation demonstrations come from automated trajectory synthesis, which composes skills such as grasp, place, handover and insert and plans motions with cuRobo v2, or from VR teleoperation with Meta Quest or Pico controllers.^[1]

For the real world, the team collected 100 demonstrations per task from four operators using a leader-follower teleoperation rig of the same embodiment, giving 1,800 trajectories (1,611,841 frames, 17.91 hours at 25 Hz) across the 18 tasks.^[1] Each demonstration has robot states, language annotations and RGB video from one head camera and two wrist cameras.^[1]

Episodes and scoring

In simulation, each task is run for 50 episodes per random seed (25 standard and 25 randomized for Generalization tasks), so a full pass is 2,100 episodes. Most policies are trained or evaluated with three seeds, giving 150 trials per task (6,300 across the 42 tasks by simple multiplication).^[1] RoboDojo reports both success rate (binary completion) and an average score that credits partial progress, and it computes the overall result as the mean of the five dimension averages rather than the mean over all tasks, so that dimensions with more tasks do not dominate.^[1] The maximum episode length is 1.2 times the 90th-percentile demonstration length (1.5 times for short synthesized tasks).^[1]

In the real world, each policy gets 10 trials per task, 180 in total, with one trained seed per embodiment.^[1] Before each trial a pre-recorded layout is replayed so that every policy starts from the same scene. Every trial is filmed and scored independently by three evaluators under a double-blind protocol; the score counts both final success and intermediate sub-steps, and the trial score is the average of the three evaluators.^[1] The appendix adds that samples with large scoring discrepancies are filtered out before averaging, and that the evaluation manager can stop a trial if the robot hits the table, the frame or itself.^[1] RoboDojo publishes the evaluation videos and scores for leaderboard entries and accepts appeals of scoring errors.^[1]

RoboDojo-RealEval

RoboDojo-RealEval is the standardized physical rig. It uses a fixed-height 1.2 m by 1.2 m white table inside a 1.5 m by 1.5 m by 2.1 m frame with black curtains and three fixed linear LED lights, custom mounts that fix the positions of the two arms and the head camera, two Gemini 305 wrist cameras and a Gemini 335L head camera, and a touchscreen web interface for scene reset, policy execution, emergency stop, video capture and cloud scoring.^[1] During reset, a semi-transparent image of the target layout is overlaid on the live camera feed so the operator can line objects up; in the authors' test, restoring a five-object scene took 14 seconds on average.^[1] The authors open-sourced the hardware setup and evaluation specifications and offer remote evaluation, in which a participant runs a policy server and connects it to the official evaluation client.^[1]

XPolicyLab

XPolicyLab is the policy-side half of the system. It converts RoboDojo trajectories into the formats different models expect (such as HDF5 files or LeRobot-style datasets), supplies training templates and deployment scripts, and defines a shared observation-action interface so that one policy implementation can be trained in simulation and then deployed to the real rig with little change.^[1] The paper integrated 30 policies through it.^[1] XPolicyLab has its own arXiv report (2608.09892, August 2026), and its GitHub repository, released under Apache 2.0, describes it as "Involving over 50 Advanced Manipulation Policies"; its README says the ecosystem integrated 69 robot policies as of October 2026.^[6][7] The RoboDojo repository itself is MIT-licensed and describes this release as "eval-only," leaving policy integration to XPolicyLab.^[5] As of 1 November 2026 the RoboDojo repository had 806 GitHub stars and 116 forks, and XPolicyLab had 513 stars and 177 forks.^[5][6]

Leaderboard rules and governance

The paper and website say the leaderboard is maintained by AI MMLab Club, which they describe as a non-profit foundation, with evaluation to be co-run by academic partners, and that no commercial company is involved in the official evaluation through sponsorship, funding or operations.^[1][4] The repository README extends this to compute, hardware and other project support.^[5]

To appear on the official verified leaderboard, a submission must meet five conditions:^[1][4]

1. It is evaluated through RoboDojo's official online evaluation system, so scores are computed by the organizers rather than self-reported.
2. Simulation results cover three random seeds (mean and standard deviation), and real-world results cover all three robots.
3. It passes a check on hidden verification layouts. If hidden-layout performance differs significantly from public-layout performance, the entry is treated as invalid.
4. The evaluated checkpoint, training and deployment code, configuration files and reproduction instructions are released through XPolicyLab when the result is published.
5. Evaluation videos are released.

Results that do not release these artifacts go on a separate closed-source track marked as non-official and unverified.^[1] The protocol page, dated 30 October 2026, adds eligibility rules: a new model must show architectural or methodological novelty or a distinct data or pre-training approach, and must have published, or plan to publish, a paper or technical report.^[4] Each entry must be one fixed system using the same checkpoints and inference procedure on every task. An agent or planner inside that system is allowed, but routing between different policies by task is not.^[4] On the simulation leaderboard, entries typed as "Agent" are listed in a separate agent leaderboard and shown unranked in the main table.^[3]

Results

Paper results (frozen 3 July 2026)

The paper froze both leaderboards on 3 July 2026.^[1] In simulation, the authors fine-tuned most of the 30 policies from each model's foundation checkpoint on RoboDojo's data (ACT was trained from scratch), with model-specific batch sizes and step counts listed in the appendix.^[1] Hy-Embodied-0.5-VLA from Tencent ranked first with an average score of 13.07 and an 8.80% success rate, followed by Spatial Forcing (12.38 / 8.04%), π0.5 (11.41 / 6.91%) and X-VLA (10.13 / 6.52%). Expert human teleoperators scored 80.42 with a 76.03% success rate; the paper notes that these experts each had more than 1,000 hours of simulation teleoperation experience and had helped build the simulation dataset.^[1] Several models, including RDT-1B, TinyVLA and OpenVLA-OFT, averaged below 1 point.^[1]

On the real robots, the paper evaluated 10 policies. π0.5 led with an overall score of 22.9 and a 12.8% success rate, while human teleoperators, each with more than 1,500 hours of real-robot teleoperation experience, completed every task (100 / 100%).^[1] The paper notes that π0.5, InternVLA-A1 and GalaxeaVLA scored well above their success rates, meaning they often made partial progress without finishing tasks.^[1]

Live leaderboard (as of 1 November 2026)

The real-world leaderboard listed 13 models at its 30 October 2026 update. The three newest entries, all world action models submitted by their developers, moved ahead of the paper's baselines.^[3]

Per-robot columns are average scores. "RoboDojo Team" marks baselines the organizers trained themselves.^[3]

The simulation leaderboard listed 66 models: 56 ranked entries and 10 agent-track entries.^[3] Most of the growth came from developer submissions after launch. At the 30 October update, the top ranked entries were WALL-OSS-1.0-lite from X Square Robot (42.59 / 34.97%), VPP2 from RobotEra (39.26 / 32.26%) and Awomo-0.5 (35.34 / 29.64%).^[3] Xiaomi's Xiaomi-Robotics-1, added on 13 July 2026, scored 20.07 / 13.93%, above the 13.07 that had led at launch.^[3] The highest agent-track scores were Q0-Preview from Qoo Tech (74.20 / 68.87%), GO-3 Nano Preview from AgiBot (61.90 / 56.42%) and PyRUA-Lean from PKU DAGroup (55.26 / 50.35%).^[3] The organizers also entered general-purpose models as agents: GPT-6 Astra scored 28.97 / 22.48%, while entries listed as GPT-5.5 and DeepSeek-Flash scored 1.13 and 2.99 (the latter over 10 episodes per task).^[3]

Findings

The paper draws six simulation findings and three real-world findings from its July results.^[1]

• No balanced generalist. Leading policies sit in a low-score regime and each is strong on different dimensions: Spatial Forcing on Generalization, X-VLA on Precision, Hy-Embodied-0.5-VLA on Long-Horizon, Memory and the overall average.^[1]
• Randomized scenes cause a broad collapse. Hy-Embodied-0.5-VLA scored 21.98 in standard Generalization scenes but 1.57 in randomized ones, a 92.9% relative drop. Spatial Forcing, which is built on π0.5, cut the relative drop from 72.2% to 67.2%, and its randomized-scene score of 6.98 was the highest of any policy; the authors credit its explicit 3D spatial grounding.^[1]
• Long-horizon tasks show partial progress. Scores well above success rates indicate that policies often complete early stages but fail to finish.^[1]
• Precision is a bottleneck. In insert_key, many policies kept executing the insertion motion after failing to hand over or align the key, which the authors read as open-loop execution without state checks. They also saw action jitter in both end-effector and joint-space policies.^[1]
• Memory structure alone is not enough. EventVLA's explicit memory helped, but many failures came from downstream manipulation errors rather than forgotten cues.^[1]
• Open tasks remain largely unsolved. The best Open result was π0.5's 1.67% success rate.^[1]
• Simulation and real-world rankings only partly agree. Among the policies tested in both settings, π0.5 led in both, but InternVLA-A1 and GalaxeaVLA did better on real robots than their simulation ranks suggested.^[1]
• Real robots expose unsafe behavior. The paper reports that DM0 frequently produced unstable control signals that required close monitoring or safety intervention, a failure that on hardware carries physical risk rather than a lower score.^[1]

Efficiency and repeatability

On 8 RTX 4090 GPUs, heterogeneous parallel simulation ran at 77.4 interactions per second with zero actions, against 40.0 for the non-heterogeneous setup (1.94 times faster), and at 64.0 against 39.2 interactions per second with π0.5 inference running on a separate A800 server (1.63 times).^[1] A full real-world pass of π0.5 over all 18 tasks (180 trials, including resets) took 202.0 minutes, about 11.2 minutes per task.^[1]

For repeatability, the authors ran three policies with three seeds on each of three RTX 4090 GPUs and found a cross-GPU standard deviation of at most 0.5 percentage points in overall success rate and 0.49 in overall score.^[1] Three repeated real-world rounds of π0.5, InternVLA-A1 and GalaxeaVLA gave standard deviations of at most 1.3 percentage points in overall success rate and 1.2 points in overall score, though single tasks varied more; store_in_safe had a success-rate standard deviation of 23.1 points for π0.5.^[1]

Comparison with other benchmarks

The paper's comparison table lists RoboDojo as the only benchmark among those compared that combines simulation and real-world evaluation, heterogeneous parallel simulation and bimanual tasks, with 24 skills, 60 tasks and "35+" integrated policies.^[1]

Selected rows from the paper's Table 8, as compiled by the RoboDojo authors.^[1]

Limitations and scope

Several limits follow from RoboDojo's design; most are acknowledged in the paper itself:

• Small real-world sample. Each policy gets 10 trials per real task and, for the paper's baselines, one trained seed per robot. The authors acknowledge larger task-level variance on contact-rich and multi-stage tasks.^[1]
• Organizer fine-tuning. Baseline policy entries (listed as contributed by "RoboDojo Team", other than the general-purpose agent entries) were fine-tuned by the RoboDojo authors, with help from some model developers acknowledged in the paper, on 100 demonstrations per task using the training settings given in the paper, not by each model's developers, so results can differ from what a model's own team might achieve. Later entries such as InternW0-Δ were submitted by their developers.^[1][3]
• No matched sim-to-real tasks. Because the real and simulated tasks differ, RoboDojo cannot measure sim-to-real transfer directly, a choice the authors made on purpose.^[1]
• Fixed hardware. All tasks use parallel-gripper bimanual arms. The authors list dexterous hands, humanoid whole-body manipulation, tactile sensing and mobile manipulation as future extensions.^[1]

RoboDojo-RC adaptation

In September 2026 Robocurve published RoboDojo-RC Tier 1, which it describes as an adaptation of six RoboDojo real-world tasks run on I2RT YAM arms with its Inspect Robots framework.^[11] Robocurve says it "evaluated Astra [GPT-6 Astra] on the 18 real-world RoboDojo tasks and selected the six easiest tasks with the highest scores," and that it uses the name RoboDojo-RC "to distinguish it from the original benchmark" because it made small changes to the physical setups and instructions and wrote its own progress rubrics.^[11] The six tasks are Pack And Pour Fruit, Stack Bowls, Store In Safe, Cap Pen, Stand Up Bottles and Sort Objects; Sort Objects uses the instruction of RoboDojo's classify_objects task.^[1][11] Robocurve's September report used a human grader for each trial rather than RoboDojo's three blinded raters, and Robocurve states that its scores "are not directly comparable with the original RoboDojo benchmark" because it adapted the setups, instructions, embodiment and rubrics.^[11] RoboDojo-RC and its leaderboard, including results for models such as GPT-6 Astra and Dream-1, are covered in the Robocurve article.

Reception

The University of Hong Kong publicized RoboDojo in an August 2026 press release that called it "the First Hong Kong-Led Unified Benchmark for Embodied AI." The release says the project was co-initiated by Professor Ping Luo and his PhD student Tianxing Chen with researchers from nearly 20 universities.^[8] The same release says the launch drew more than 100,000 views on X in its first week and that RoboDojo's open-source resources passed 100,000 downloads on Hugging Face; these figures come from HKU and were not independently checked.^[8] QbitAI (8 July 2026, an authorized republication whose views it attributes to the original author) and Jiqizhixin (17 July 2026) both carried launch articles emphasizing the gap between the best models and human experts.^[9][10]

Authors and affiliations

The paper lists 44 authors and 18 affiliations: HKU MMLab, UC Berkeley, Tsinghua University, Peking University, Stanford, MIT, UNC, Princeton, Carnegie Mellon, the National University of Singapore, Nanyang Technological University, the Chinese University of Hong Kong, Imperial College London, Shanghai Jiao Tong University, Northwestern University, HKUST (Guangzhou), Zhejiang University and Yale.^[1][2] Tianxing Chen and Yue Chen are co-project leaders, and Yue Chen, Mingyu Ding, Wenbo Ding, Ping Luo and Masayoshi Tomizuka are the corresponding authors.^[1][2]

Continue reading

• RRobocurveRobocurve (Robocurve, Inc., which styles itself Robocurve PBC) is a San Francisco public benefit corporation that evaluates how well frontier AI models and robot-control models perform on real robots, and…Mentioned 6 times above
• RRoboChallengeRoboChallenge is an online benchmarking platform that evaluates robot control policies, mainly vision-language-action models (VLAs), on physical robots instead of in simulation.Mentioned twice above
• LLIBEROLIBERO ("LIfelong learning BEnchmark on RObot manipulation tasks") is an AI benchmark for studying knowledge transfer in lifelong robot learning.Mentioned twice above
• VVision-language-action modelA vision-language-action model (VLA model, or VLA) is a machine-learning model for robotics that uses visual observations and a natural-language instruction to generate actions a robot can execute.Listed in See also
• RRobot foundation modelA robot foundation model is a large-scale machine learning model, typically based on the transformer architecture, that is pre-trained on broad, diverse datasets of robot interactions and then adapted to a…Listed in See also
• RRoboArenaRoboArena is a distributed, real-world benchmark for generalist robot policies in which volunteer evaluators at different laboratories run double-blind, pairwise (A/B) comparisons of policies on tasks and…Listed in See also

More from See also

• RoboCasa
• World action model
• Imitation Learning

More to read

• NVIDIA Isaac Sim
• GPT-6 Astra
• Xiaomi (robotics)

Topics:AI BenchmarksEmbodied AIModel EvaluationRobotics

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49 ^50 ^51 ^52 ^53 ^54 ^55 ^56 ^57 ^58 ^59 ^60 ^61 ^62 ^63 ^64 ^65 ^66 ^67 ^68 ^69 ^70 ^71 ^72 ^73Chen, Tianxing, Yue Chen, et al. "RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies". arXiv:2607.04434, submitted 5 July 2026 (v3, 8 July 2026).
2. ^1 ^2 ^3 ^4RoboDojo. "RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies". Project homepage. Accessed November 1, 2026.
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10RoboDojo. "Leaderboard". RoboDojo-Sim and RoboDojo-RealWorld leaderboards, updated October 30, 2026. Accessed November 1, 2026.
4. ^1 ^2 ^3 ^4 ^5RoboDojo. "Evaluation Integrity and Anti-Gaming Protocols". Dated October 30, 2026. Accessed November 1, 2026.
5. ^1 ^2 ^3 ^4RoboDojo-Benchmark. "RoboDojo". GitHub repository. Accessed November 1, 2026.
6. ^1 ^2 ^3 ^4XPolicyLab. "XPolicyLab". GitHub repository. Accessed November 1, 2026.
7. ^XPolicyLab Community, Tianxing Chen, Yue Chen, et al. "XPolicyLab: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment". arXiv:2608.09892, August 10, 2026.
8. ^1 ^2 ^3 ^4 ^5The University of Hong Kong. "HKU Pioneers 'RoboDojo': The First Hong Kong-Led Unified Benchmark for Embodied AI, Setting New Global Standards for Robot Reliability". Press release, August 12, 2026.
9. ^1 ^2QbitAI. "具身智能'高考'难疯了！人类100分，最强模型12.8". WeChat, July 8, 2026.
10. ^1 ^2Jiqizhixin (Synced). "RoboTwin原班团队再下场，构建具身评测珠峰". WeChat, July 17, 2026.
11. ^1 ^2 ^3 ^4 ^5Machcha, Sravanthi, Achu Menon, Sabrina Zou, Tzu Kit Chan, and Jay Chooi. "Opus 5.5 on RoboDojo-RC Tier 1: higher scores at lower cost". Robocurve, September 23, 2026.

Show all 11 references