Zero-HLM

Zero-HLM (full title: Zero-HLM: Generalizable Humanoid Loco-Manipulation without Real Robot Data) is a real-to-sim-to-real research framework for training humanoid robot loco-manipulation policies on simulation data alone. It was described in a 12-author paper posted to arXiv on 26 October 2026 by researchers from the Beijing Institute of Technology, the State Key Laboratory of General Artificial Intelligence at BIGAI, Tsinghua University and Delta Intelligence.^[1][2] The system rebuilds a real room in simulation from smartphone captures, expands 45 teleoperated simulation demonstrations into 4,800 physically validated whole-body demonstrations, and trains visuomotor policies that the authors deploy zero-shot on a Unitree G1 humanoid.^[2]

The paper's headline results are author-reported. On three long-horizon tasks, NVIDIA Isaac GR00T N1.7 fine-tuned on 800 Zero-HLM demonstrations per task reached 66.7% mean full-task success on the real robot, against 46.7% for the same model fine-tuned on 100 real teleoperated demonstrations per task. Across five generalization settings covering shifted real-world configurations, pooled success rose from 23% for the real-data policy to 69% for a policy trained on expanded simulation data, without any additional real demonstrations.^[1][2] The paper also introduces Zero-HLM-Bench, a simulation benchmark with three tasks and nine skill stages.^[2]

Background

Humanoid loco-manipulation combines walking with object manipulation, so a single task can require the robot to walk to a table, squat, carry an object and use its legs to move furniture. The Zero-HLM authors argue that learning such behaviors in a way that generalizes is bottlenecked by the cost of collecting diverse demonstrations on a physical humanoid.^[2] Simulation is the usual alternative, but sim-to-real transfer for humanoids raises two problems the paper treats as coupled: building scene-scale environments that are both visually faithful and usable by a physics engine, and generating successful demonstrations under one unified whole-body controller.^[1][2]

Earlier real-to-sim-to-real work built on Gaussian splatting to reduce the visual gap between rendered and real camera images, and the paper notes that such methods had shown promising transfer in tabletop settings. Extending the approach to a whole room is harder, because meshes extracted from Gaussian reconstructions can contain contact artifacts and articulated objects such as cabinet doors need extra modeling.^[2] On the data side, MimicGen showed that a few source demonstrations can be adapted to new object configurations through simulation rollouts.^[2][8] Several humanoid data generators separate locomotion commands from upper-body commands; the Zero-HLM authors argue this separation limits natural motion and leg-based interaction, which motivates using a unified whole-body motion tracker instead.^[2] The reconstruction step is also inspired by agentic "digital cousin" pipelines that use multimodal language models to assemble simulated scenes resembling a real one.^[2][9]

The paper's comparison table positions Zero-HLM against DemoHLM, VIRAL, FetchMan, HumanoidMimicGen, LEGS, OASIS and VLK. By the authors' own accounting, Zero-HLM is the only one of these that combines a unified whole-body policy, automated physics-ready scene reconstruction, few-demonstration skill adaptation, photorealistic rendering, articulated tasks, leg-critical interaction and zero-shot deployment.^[2] VLK, for example, also uses unified whole-body control in reconstructed scenes, but the Zero-HLM paper describes it as focusing mainly on single-object transport.^[2][10]

Method

The framework has four parts, which the project page lists as agentic real-to-sim, source demonstrations, scalable data generation and direct sim-to-real deployment.^[3]

Gaussian-guided agentic real-to-sim reconstruction

The scene is reconstructed from smartphone images with MAtCha Gaussians, a sparse-view Gaussian reconstruction method published at CVPR 2025, while checkerboards fix metric scale and ground alignment.^[2][6] The paper's reconstruction figure shows a kitchen-style room captured in more than 200 photos.^[2] A multimodal agent then uses the Gaussian reconstruction and a mesh extracted from it to build and configure the physical scene through Blender MCP and Isaac Lab. The extracted mesh guides initial layout, and the agent picks inspection viewpoints and compares Gaussian and mesh renderings from matched camera poses to refine local geometry and object placement.^[2]

Objects that the robot touches are reconstructed separately from dedicated video captures, with camera poses estimated by COLMAP. Inside Isaac Lab, the agent sets collision geometry, articulated joints, mass, friction and center of mass, then checks them with interaction tests such as opening a cabinet and dropping an object onto a surface.^[2] The result is a hybrid digital twin with a mesh-based physics backend and a Gaussian visual frontend.^[1][2]

The paper's text calls the reconstruction agent "a multimodal agent" without naming the model. Siyuan Huang's announcement post said the team builds "visually faithful, physics-ready digital twins with Astra" and asked whether "GPT-Astra" could "boost sim2real capability".^[2][4] Separately, the paper uses OpenAI's GPT-6 Astra as an agentic data-generation baseline (see below).^[2]

Motion-environment co-adaptation

Each of the three tasks starts from 15 successful teleoperated simulation demonstrations. Each demonstration is segmented into locomotion, manipulation and leg-critical interaction using event predicates based on hand closure, object proximity, displacement and task success.^[2] For each newly sampled configuration, the system picks one of the nearest valid source demonstrations by position and rotation distance.^[2]

In Stage 1, A* plans collision-free walking paths and the SONIC motion planner turns them into whole-body reference motion at a walking speed sampled from 0.5 to 0.7 m/s. Manipulation segments keep the demonstrated hand-to-object relation by transforming the source hand pose into the new object frame and solving arm inverse kinematics. Leg-critical segments, such as kicking a door shut, keep the demonstrated lower-body motion. A decaying hand-position offset and speed-based retiming join the segments together.^[2]

In Stage 2, each reference is executed in simulation through SONIC, NVIDIA's whole-body motion tracker: its encoder maps a window of reference motion to a 64-dimensional latent command, and its decoder turns that latent plus proprioceptive history into joint targets for PD control. The two hands are commanded separately.^[2] Rollouts that fall, collide invalidly or fail beyond correction are resampled.

Stage 3 handles the paper's central difficulty: even a kinematically valid reference can be tracked imperfectly, so the robot may arrive slightly off its intended pose and miss a grasp. Instead of only rejecting such rollouts, Zero-HLM moves the environment to match the robot. A planar SE(2) transform shifts relevant scene assets to restore their intended relation to the executed robot pose; an object-level correction shifts the object or placement target by the residual tracking error relative to a successful source rollout; and a Z-axis compensation adjusts the reference wrist height around grasp closure.^[2] Corrected rollouts are re-validated under the same criteria.

Visual observations and deployment

Training images come from the physically validated executions, not from the planned references. The system replays recorded robot and object states and composites Gaussian background renders with simulator-rendered robots and objects through mask-guided z-buffer compositing, producing egocentric images for policy training.^[2] At deployment, the policy predicts SONIC motion latents that a frozen SONIC decoder converts into joint targets, while hand commands go through a separate interface, the same split used during data generation.^[2]

Experimental setup

Source: Zero-HLM paper, Section 4.1.^[2]

The three evaluation tasks are:^[2][3]

The authors define a Base configuration, which matches where real demonstrations were collected, and Expanded configurations that shift robot start positions and picking or placement locations for Tasks A and B. Task A also has an alternative object-goal pair (a bitter melon placed on a white plate).^[2] The 4,800 demonstrations break down as 800 Base demonstrations for each of the three tasks, 800 Expanded demonstrations for each of Tasks A and B, and 800 object-goal demonstrations.^[2] Each policy was tested on the physical robot in 20 trials per task under Base and five trials per configuration under Expanded.^[2]

Results

All figures in this section are reported by the authors and have not, as of 30 October 2026, been independently reproduced.

Reconstruction and data generation

Building the scene, configuring physics and reconstructing seven interaction assets took 7 hours 13 minutes in total, at an allocated subscription cost of US$22.82.^[2]

Against a ground-truth scan from a handheld LiDAR scanner, the full agentic reconstruction reached a symmetric Chamfer distance of 5.517 cm in the table and cabinet regions, compared with 5.894 cm for a mesh extracted directly from the Gaussians (GS-Mesh) and 11.689 cm for the agent working without Gaussian guidance.^[2]

Zero-HLM's generation pipeline succeeded on 33%, 34% and 25% of attempts for Tasks A, B and C. Collecting 50 successful demonstrations took 0.56, 0.63 and 1.19 hours in simulation, against 1.35, 1.02 and 1.68 hours of real-world teleoperation (excluding setup), which the authors report as speedups of 2.4x, 1.6x and 1.4x, averaging 1.8x.^[2]

An ablation on Task A showed which corrections matter most. With all components, 61% of generation attempts succeeded; removing velocity-adaptive transitions, Z-axis compensation or object-level correction lowered this to 55%, 51% and 49%, and removing the robot-pose-guided scene correction dropped it to 16%.^[2] A separate real-robot test with 100 simulated demonstrations compared rendering methods: Gaussian-rendered training images raised Task A full-task success from 3/20 with mesh renders to 7/20, with picking improving from 3/20 to 10/20.^[2]

Direct sim-to-real transfer

Under the Base configuration, the authors fine-tuned pretrained GR00T N1.7, a member of NVIDIA's GR00T N1 family of humanoid foundation models,^[7] on four data sources and ran each policy on the real G1 through the same SONIC controller.^[2]

With matched demonstration counts (100 per task), Zero-HLM data performed close to real data (43.3% versus 46.7%). With 800 simulated demonstrations per task, mean success rose to 66.7%, a 20-percentage-point gain over the real-data policy.^[2]

Generalization across configurations

To test configurations not covered by the real demonstrations, the authors compared Real-Only(100) with a policy trained on 800 demonstrations that included Expanded configurations. Pooled over five settings and 100 real trials, success rose from 23/100 to 69/100.^[2]

The object-goal setting is language-conditioned: the same policy is told either "Place the banana in the woven basket" or "Place the bitter melon on the white plate" with both objects present.^[3] The project page also shows an uncut recording of 18 walk-pick-and-place attempts, of which 11 ended in a successful placement.^[3]

What the transfer analysis shows

The paper includes three diagnostic studies on Task A and Tasks A-C:^[2]

• Simulation as a predictor. In Zero-HLM-Bench and on the real robot, a real-data policy scored 31% in simulation and 40% in reality, while a simulation-data policy scored 39% and 35%. The absolute ranges were close, but the ranking of the two policies reversed, so the authors describe the benchmark as an approximate measure of task difficulty rather than an exact predictor of policy rankings.
• Actions versus images. In a picking-region study, mixing 25 real and 25 simulated demonstrations reached only 2/5 and 4/5 in the two regions. Replaying the simulated trajectories on the real robot and swapping in real camera images, while keeping the simulation action labels, reached 5/5 in both. The authors read this as support for the transferability of simulated action labels and as evidence that the visual gap remains a limitation.
• Co-training. Adding 400 or 800 simulated trajectories to 100 real ones gave mean full-task success of 46.7% and 53.3%, both below simulation-only training with 800 demonstrations (66.7%).

Zero-HLM-Bench

Zero-HLM-Bench runs the three tasks in Isaac Lab with a simulated G1, 640x480 egocentric RGB input, the same 34-dimensional state and the same 66-dimensional SONIC-latent action space for every model. Rollouts run at 50 Hz for at most 2,000 control steps (40 seconds), and each model-task pair is evaluated on 100 trials. Stage-wise scores are measured independently of whether earlier stages succeeded.^[2] All models were trained on the same 800 generated trajectories per task.^[2]

GR00T N1.7 had the highest mean, while DiT4DiT was best on Task B. The authors point out that stage success does not guarantee task success: Diffusion Policy walked successfully in 90% of Task A trials but completed the full task in 3%.^[2] For GR00T N1.7, success rose with more generated data on all three tasks, with larger gains at small data scales.^[2]

GPT-6 Astra baseline

To compare against a general-purpose agent, the authors gave GPT-6 Astra the same reconstructed scenes, assets, task configurations and source demonstrations, plus full simulator state and access to the SONIC planner, inverse kinematics and whole-body control interfaces, and asked it to design its own data-generation pipeline. The appendix reproduces the full Task B prompt.^[2] The authors report results only after "human-assisted debugging and task alignment".^[2]

After that stabilization, the Astra-built pipelines succeeded on 21.38% of Task A attempts and 24.70% of Task B attempts, and produced no successful Task C demonstrations after more than eight hours. Its trajectories often contained pauses and repeated attempts and frequently exceeded one minute, compared with about 30 seconds for real demonstrations; the authors say policies trained on them showed similar hesitation on the real robot.^[2] On the real G1, the Sim-Astra(100) policy averaged 10.0% full-task success on Tasks A and B.^[2]

Authors and affiliations

Dianyi Yang and Mengya Liu are listed as equal-contribution first authors. Wei Liang, Baoxiong Jia and Siyuan Huang are corresponding authors, and Baoxiong Jia is listed as project leader.^[1][3]

BIGAI is the Beijing Institute for General Artificial Intelligence. Delta Intelligence is a humanoid-robot foundation-model company that, according to Gasgoo, was founded in January 2026 as a spin-out of BIGAI.^[5] Several of the same researchers (Yixuan Li, Le Ma, Mengya Liu, Wei Liang, Baoxiong Jia and Siyuan Huang) also co-authored OmniClone, a whole-body humanoid teleoperation system that the Zero-HLM paper cites as related work.^[2]

Limitations

The authors identify the remaining visual-domain gap as an important limitation: their action-versus-image study and co-training results suggest that mismatched images, rather than simulated action labels, limit the benefit of mixing real and simulated data.^[2] The benchmark agrees with real-world results only approximately and reversed the ranking of two policies.^[2] All reported experiments use one robot (the Unitree G1) and one whole-body controller (SONIC). The authors list extending the framework to other unified whole-body controllers, studying how tracking accuracy affects demonstration quality, and extending the benchmark to richer interaction skills and multiple reconstructed scenes as future work.^[2] Generation still starts from 15 teleoperated source demonstrations per task, and the GPT-6 Astra comparison reflects a pipeline that required human-assisted debugging.^[2] Real-world evaluation used 20 trials per task under Base and five trials per Expanded configuration.^[2]

Continue reading

• SSONIC (Humanoid Control)SONIC, short for Supersizing mOtion tracking for Natural humanoId Control, is a physics-based whole-body motion tracking and control system developed by Nvidia.Mentioned 14 times above
• SSim-to-real transferSim-to-real transfer (also written as sim2real) is the process of training a policy, controller, or perception model inside a physics simulator and deploying it on a physical robot in the real world without…Mentioned twice above
• WWhole-body controlWhole-body control (WBC) is a class of robotics control techniques that coordinates all of a robot's degrees of freedom at once to achieve multiple prioritized tasks, such as balancing, reaching, and…Mentioned twice above
• GGaussian SplattingGaussian splatting is a method for real-time radiance field rendering that represents a 3D scene as a collection of millions of anisotropic 3D Gaussian primitives, each defined by a position, covariance…Listed in See also
• NNVIDIA Isaac GR00T N1.7NVIDIA Isaac GR00T N1.7 is a 3-billion-parameter, cross-embodiment vision-language-action model developed by NVIDIA for humanoid and manipulation robots.Listed in See also
• SSynthetic dataSynthetic data is data that is artificially generated by algorithms, simulations, or generative models rather than collected from real-world events, used as a substitute for or supplement to real data when…Listed in See also

More from See also

• Embodied AI

More to read

• Unitree G1
• arXiv
• Tsinghua University
• Humanoid Robot
• Digital twin

Topics:AI ResearchEmbodied AIHumanoid RobotsRobotics

References

1. ^1 ^2 ^3 ^4 ^5Yang, D., Liu, M., Wang, Z., Liu, Y., Zhang, X., Li, Y., Ma, L., Yang, Y., Huang, W., Liang, W., Jia, B., and Huang, S. "Zero-HLM: Generalizable Humanoid Loco-Manipulation without Real Robot Data." arXiv:2610.35075 [cs.RO], submitted 26 October 2026. arxiv.org/...2610.35075
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49 ^50 ^51 ^52 ^53 ^54Yang, D., et al. "Zero-HLM: Generalizable Humanoid Loco-Manipulation without Real Robot Data" (full text, v1). arXiv HTML. arxiv.org/...2610.35075v1
3. ^1 ^2 ^3 ^4 ^5Zero-HLM project page. young-bit.github.io/zero-hlm
4. ^Huang, Siyuan (@siyuanhuang95). Announcement post for Zero-HLM, X, 29 October 2026. x.com/...2115642053673775220
5. ^Gasgoo. "Seeds | Backed by AGIBOT, Leju, and Galaxea AI, who is Delta Intelligence?" 29 July 2026. autonews.gasgoo.com/...ligence-2082453965206691841
6. ^Guedon, A., Ichikawa, T., Yamashita, K., and Nishino, K. "MAtCha Gaussians: Atlas of Charts for High-Quality Geometry and Photorealism from Sparse Views." CVPR 2025. openaccess.thecvf.com/...torealism_CVPR_2025_paper
7. ^NVIDIA, et al. "GR00T N1: An Open Foundation Model for Generalist Humanoid Robots." arXiv:2503.14734, 2025. arxiv.org/...2503.14734
8. ^Mandlekar, A., et al. "MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations." arXiv:2310.17596, 2023. arxiv.org/...2310.17596
9. ^Dai, T., et al. "Automated Creation of Digital Cousins for Robust Policy Learning." arXiv:2410.07408, 2024. arxiv.org/...2410.07408
10. ^Wang, Y.-J., et al. "VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes." arXiv:2606.30645, 2026. arxiv.org/...2606.30645

Show all 10 references