# Hydra-0

> Source: https://aiwiki.ai/wiki/hydra_0
> Updated: 2026-09-03
> Fact-checked: 2026-09-03
> Categories: AI Models, NVIDIA, Robotics
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Hydra-0." aiwiki.ai, 3 Sept 2026. https://aiwiki.ai/wiki/hydra_0
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

Hydra-0 is an experimental [world model](https://aiwiki.ai/wiki/world_model) for robot manipulation that represents actions as trajectories of visible points in an image. Its authors call this representation "action flow." Instead of giving a video model joint angles or end-effector commands that apply to one robot design, Hydra-0 describes how points on a robot, gripper, hand, or manipulated object move across the camera plane. The system was presented in an August 2026 technical report by researchers affiliated with [NVIDIA](https://aiwiki.ai/wiki/nvidia), Brown University, Columbia University, and Harvard University.[1][2]

Hydra-0 is a research system rather than a single publicly downloadable checkpoint. The paper reports variants built on Cosmos 2.5 and Wan2.2 video models, along with forward prediction, open-loop policy evaluation, and one inverse-control proof of concept. As of September 3, 2026, the project page labeled the code as "coming soon" and did not provide Hydra-0 checkpoints or a software license.[1][2]

## Publication and scope

| Field | Detail |
|---|---|
| Full title | Hydra-0: Action Flow for Generalist World Modeling and Control |
| Publication status | Version 1 arXiv technical report submitted August 18, 2026 |
| Authors | Hongyu Li, Bowen Wen, Xinghao Zhu, Yixuan Wang, Yilun Du, Yunzhu Li, George Konidaris, Stan Birchfield, Soha Pouya, Chenran Li, and Yan Chang |
| Affiliations listed | NVIDIA, Brown University, Columbia University, and Harvard University |
| Primary domain | Robot manipulation, especially interaction with deformable objects |
| Core interface | Sparse image-plane point trajectories with visibility flags |
| Video backbones tested | Cosmos 2.5 2B, Wan2.2 I2V-A14B, and Wan2.2 TI2V-5B |
| Public artifacts on September 3, 2026 | Paper and interactive project page; code listed as coming soon |

The name refers to the action-flow system described in arXiv:2608.18077. The cited report does not claim that Hydra-0 is a general robot policy that can directly control arbitrary hardware. It evaluates a predictive model, an open-loop replay method, and a separately adapted world action model.[1]

## Why use image-space actions

Robot commands are usually tied to an embodiment. Joint-space commands encode a particular kinematic chain, while the same end-effector command can produce different visible link motion on robots with different geometry. A video predictor conditioned directly on those commands must learn both the scene dynamics and an embodiment-specific mapping from commands to pixels.[1]

Hydra-0 moves that interface into the observation itself. For each tracked point, action flow records a two-dimensional image position and a visibility flag at every step in a prediction horizon. The same data structure can therefore describe motion from human hands, handheld grippers, single-arm robots, and bimanual systems. The representation does not erase physical constraints. At deployment, the method first executes a candidate command in [NVIDIA Isaac Lab](https://aiwiki.ai/wiki/isaac_lab), then projects the resulting robot-surface motion through a calibrated camera.[1][2]

The authors describe two ways to construct action flow:

| Route | When it is used | How the trajectories are obtained |
|---|---|---|
| Geometry-aware | Metadata-rich training data and forward deployment | Visible surface points are attached to robot links, moved with recorded or simulated link transforms, and projected through camera intrinsics and extrinsics. A depth-buffer test removes self-occluded points. |
| Video-only | Training videos without robot descriptions or calibration | Dense point tracks are recovered from video, then grounded masks divide tracks among the acting embodiment, manipulated object, and other scene content. |

The video-only route uses AllTracker for dense point tracking.[1][8] Its observed future trajectories are training labels, not information available to the deployed predictor. In forward deployment, action flow is constructed causally from a candidate executable command and known robot geometry. Large camera-calibration errors can weaken the alignment between the projected motion and the real image, and occlusions by unknown objects are not explicitly removed by the geometry-aware procedure.[1]

## Conditioning and model design

Hydra-0 samples a feature from the first encoded frame at each track's starting point. It propagates that feature along the requested trajectory with local Gaussian weights. A presence gate indicates which locations received valid trajectory information. The propagated features and gate are then supplied to the video backbone while it denoises the future latent sequence.[1]

The report implements this mechanism on [NVIDIA Cosmos](https://aiwiki.ai/wiki/nvidia_cosmos) 2.5 and two Wan2.2 configurations.[3][4] The Cosmos variant adds a separate motion-side projection to its input tokens. The Wan2.2 I2V-A14B variant uses the model's visual-conditioning pathway, while the TI2V-5B variant widens the input projection for a 49-channel side input. The pretrained backbone parameters are frozen except for the patch embedding, and rank-64 [LoRA](https://aiwiki.ai/wiki/lora) adapters are trained on attention and feed-forward projections.[1][9]

During the canonical image-to-video training mixture, the system draws one of four track-selection modes at each step:[1]

| Mode | Sampling probability | Information supplied |
|---|---:|---|
| Embodiment | 0.40 | Tracks on the acting body, such as robot links, grippers, or human hands |
| Object | 0.40 | Tracks on the manipulated object or a supplied desired object trajectory |
| All | 0.15 | A sample from embodiment, object, and unassigned scene tracks |
| None | 0.05 | No trajectory condition, leaving text and image conditioning |

These probabilities apply to the paper's canonical mixture. Dataset-specific overrides are used when a source lacks reliable track categories. If a selected category is unavailable, the method renormalizes over the remaining valid modes.[1]

## Training corpus

The authors report a seven-source corpus containing 178,187 episodes. After filtering, it contains 1,565,634 non-overlapping video windows totaling 2,201.7 hours. Videos are resampled to 480p at 16 frames per second, and a standard window contains 81 frames, or about five seconds. Filters remove selected static windows, frozen-gripper windows, and DROID episodes with contentless language annotations.[1]

| Dataset | Embodiment | Episodes | Filtered windows | Hours | License reported for source data |
|---|---|---:|---:|---:|---|
| DROID | Single arm | 70,339 | 223,075 | 313.7 | CC BY 4.0 |
| ABC-130k | Bimanual arms | 54,961 | 1,048,681 | 1,474.7 | Apache 2.0 |
| MolmoAct2 | Bimanual arms | 7,723 | 126,335 | 177.7 | Apache 2.0 |
| EgoDex | Human hands | 41,888 | 89,380 | 125.7 | CC BY-NC-ND 4.0 |
| Deform360 | Handheld grippers | 1,714 | 60,315 | 84.8 | MIT |
| XVLA-Soft-Fold | Bimanual arms | 1,524 | 17,772 | 25.0 | Apache 2.0 |
| H1-Fold-Clothes | Bimanual arms | 38 | 76 | 0.1 | Apache 2.0 |
| Total | Multiple | 178,187 | 1,565,634 | 2,201.7 | Mixed |

DROID is the largest single-arm source in the mixture and contributes varied scenes from real robot manipulation.[5] ABC-130k provides most of the filtered hours. EgoDex supplies egocentric human-hand interaction, while Deform360 and the two folding sets extend the corpus toward deformable-object manipulation. The Interactive World Simulator data used in the adaptation study is excluded from the multi-embodiment training corpus.[1]

The license column reproduces the labels reported by the Hydra-0 authors for each source dataset. Those source-data licenses do not establish a license for Hydra-0 code or checkpoints.[1]

## Forward prediction

In forward mode, Hydra-0 combines a physics simulator with a learned video predictor. Isaac Lab executes a candidate command and provides link transforms. Robot geometry and camera calibration turn those transforms into action flow. The video model receives the first real observation and that flow, then predicts how the rest of the scene may respond.[1][2]

This division assigns known rigid-body robot motion to the simulator and asks the learned component to generate the wider visual consequence, including contacts and deformable-object motion. The method still depends on the accuracy of the command rollout, robot model, and calibration. It is not a replacement for the physics stage.[1]

## Reported prediction results

The paper evaluates five held-out sets: XVLA-Soft-Fold, Deform360, DROID, MolmoAct2, and ABC-130k. Each method is evaluated on 100 clips from each set. The table below shows the paper's unweighted averages across available dataset strata. PSNR and SSIM measure full-reference image fidelity; endpoint error (EPE) measures tracked object or gripper motion in pixels; FID and FVD measure frame and video distribution differences; and the VLM score is a 1-5 automated rating.[1]

| Model in the Hydra-0 study | PSNR higher is better | SSIM higher is better | Object EPE lower is better | Gripper EPE lower is better | FID lower is better | FVD lower is better | VLM higher is better |
|---|---:|---:|---:|---:|---:|---:|---:|
| Action-conditioned Cosmos 2.5 baseline | 15.62 | 0.668 | 13.23 | 34.28 | 39.1 | 405.8 | 3.88 |
| Hydra-0 with Cosmos 2.5 2B | 18.41 | 0.725 | 6.27 | 13.80 | 32.4 | 277.4 | 3.83 |
| Hydra-0 with Wan2.2 A14B | 20.76 | 0.805 | 6.00 | 3.83 | 20.7 | 193.7 | 3.98 |
| Hydra-0 with Wan2.2 A14B, four-step student | 21.84 | 0.830 | 5.27 | 3.29 | 18.7 | 155.9 | 4.23 |

Against the action-conditioned Cosmos 2.5 baseline, the authors calculate that their best four-step Wan2.2 configuration reduces average gripper-motion error by 90.4% and average object-motion error by about 60.2%.[1] This comparison changes both the conditioning scheme and the video backbone. The more controlled comparison between the two Cosmos 2.5 rows keeps the backbone family fixed. In that comparison, the action-flow variant improves the reported image, motion, FID, and FVD point estimates, but its average automated VLM score is slightly lower, at 3.83 instead of 3.88.[1]

Object EPE is not reported for DROID because the authors could not obtain reliable manipulated-object masks in its cluttered scenes. The object-EPE average therefore covers four datasets, while other displayed averages cover five. All results in this section come from the Hydra-0 study and project page; the cited materials do not contain an independent replication.[1][2]

## Adaptation and inference speed

For data-efficiency experiments, the authors adapt models to six held-out Interactive World Simulator tasks using nested fractions from 0% to 100% of each task's eligible training set. They report that the multi-embodiment checkpoint has better zero-shot LPIPS, object-flow EPE, and FVD point estimates than the pretrained-only Wan2.2 initialization on all six tasks. Most changes in its reported metrics occur by the 20% fraction. The authors caution that FVD is estimated from 40 clips per point and that small non-monotonic changes after 20% do not establish convergence.[1]

The reported speed test isolates video generation. It uses one 80 GB H100 GPU, bfloat16, batch size one, 81 output frames at 480 by 832 pixels, no classifier-free guidance, and no VAE decoding. Each value is the median of five measured clips after three warm-up clips.[1]

| Stage | Seconds per clip | Frames per second | Speedup vs. bidirectional teacher |
|---|---:|---:|---:|
| Bidirectional teacher, 50 steps | 20.92 | 3.87 | 1.0x |
| Autoregressive teacher, 50 steps | 12.48 | 6.49 | 1.68x |
| Few-step student, four steps | 1.31 | 61.98 | 16.0x |

The 16.0x figure is a generation-only comparison with the bidirectional teacher. It excludes guidance, VAE and pixel decoding, robot simulation, tracking, and other system work. The benchmark also uses a short three-chunk horizon where the paper's longer-rollout attention cap is inactive. It is not an end-to-end robot-control speedup.[1]

## Open-loop policy evaluation

Hydra-0 is also tested as a replay-based evaluator on the RoboLab benchmark.[6] The study covers five [vision-language-action](https://aiwiki.ai/wiki/vision_language_action_model) policies across six tasks, with ten episodes for each policy-task pair, or 300 episodes in total. The policies are pi-0, pi-0.5, [NVIDIA Isaac GR00T N1.7](https://aiwiki.ai/wiki/groot_n1_7), Cosmos-3 Nano, and Cosmos-3 Edge.[1]

For each episode, Hydra-0 starts from the true first observation and receives action flow derived from the full recorded trajectory. The policy is not asked to act on generated frames. A human rater, blinded to policy identity, judges the generated rollout against the task's success criterion.[1]

The authors report a Pearson correlation of 0.96 and a Spearman correlation of 0.93 between generated and reference success rates across policy-task aggregates, with a mean absolute error of 5.7 percentage points. Per-episode verdicts agree 93.0% of the time, with Cohen's kappa of 0.82, ten false positives, and eleven false negatives.[1][2]

This protocol tests whether a generated replay preserves the outcome of an already recorded motion. It does not test prospective evaluation of a new policy acting on generated observations, and it does not measure error accumulation in a closed feedback loop.[1]

## Inverse mode and robot-control demonstration

The report adapts the same interface into a world action model. In this inverse mode, the input is desired object flow rather than robot embodiment flow. The model generates compatible visible robot motion, and a separately trained action head reads internal [Diffusion Transformer](https://aiwiki.ai/wiki/diffusion_transformer) features to produce target-robot commands. Training uses paired real-world rollouts that include successful and failed attempts, so the authors do not require task-specific expert demonstrations for the reported target task.[1]

The physical demonstration is limited to one flexible-pipe-bending task. The desired object trajectory comes from a held-out human demonstration, and the action head outputs 14-degree-of-freedom commands for a bimanual YAM robot. The reported world action model uses the 50-step autoregressive teacher, not the four-step student from the inference benchmark.[1][2]

This experiment shows one successful transfer from human-demonstrated object motion to robot execution. It is not evidence of broad closed-loop control across the training datasets or embodiments.[1]

## Relationship to earlier work

Image-space trajectories predate Hydra-0 as conditions for video generation and robot manipulation. Flow as the Cross-domain Manipulation Interface, for example, studies point flow as a common representation for manipulation across embodiments.[7] Hydra-0 combines this general idea with kinematically grounded deployment flow, multi-embodiment video training, open-loop replay, and an inverse readout in one reported system.[1]

The model variants inherit their video-generation foundations from Cosmos 2.5 and Wan.[3][4] Hydra-0 adds motion-conditioning projections and low-rank adapters rather than retraining every backbone parameter. Its technical claims should therefore be read as claims about the action-flow interface and the authors' trained variants, not as a new base video architecture.[1]

## Limitations and availability

The paper identifies centimeter-scale grasp imprecision in the inverse model and ambiguity about depth, grasp state, and contact. It offers only a qualitative wrist-camera example, and broader camera motion and mobile manipulation are not systematically evaluated. All policy evaluation is open loop. The real-robot inverse-control evidence is a single task.[1][2]

The results are reported by the system's authors in a version-1 preprint. The cited sources do not provide an independent reproduction. Several headline figures compare different backbones, and the 16.0x speed result excludes major parts of a deployed pipeline.[1]

As of September 3, 2026, the project page's code control still read "coming soon," and its footer said that code links would follow. Neither the paper nor the project page identified a Hydra-0 checkpoint license.[1][2] The arXiv submission uses arXiv's perpetual non-exclusive distribution license for the article itself, which is not a software or model license.[10]

## References

1. Hongyu Li et al., "Hydra-0: Action Flow for Generalist World Modeling and Control," arXiv:2608.18077v1, August 18, 2026. https://arxiv.org/abs/2608.18077
2. NVIDIA Isaac, "Hydra-0: Action Flow for Generalist World Modeling and Control," project page, August 2026. https://nvidia-isaac.github.io/video_to_data/hydra-0/
3. NVIDIA et al., "World Simulation with Video Foundation Models for Physical AI," arXiv:2511.00062v2, February 24, 2026. https://arxiv.org/abs/2511.00062
4. Team Wan et al., "Wan: Open and Advanced Large-Scale Video Generative Models," arXiv:2503.20314v2, April 19, 2025. https://arxiv.org/abs/2503.20314
5. Alexander Khazatsky et al., "DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset," arXiv:2403.12945v2, April 22, 2025. https://arxiv.org/abs/2403.12945
6. Xuning Yang et al., "RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies," arXiv:2604.09860, April 2026. https://arxiv.org/abs/2604.09860
7. Mengda Xu et al., "Flow as the Cross-domain Manipulation Interface," Proceedings of Machine Learning Research, volume 270, 2025. https://proceedings.mlr.press/v270/xu25a.html
8. Adam W. Harley et al., "AllTracker: Efficient Dense Point Tracking at High Resolution," arXiv:2506.07310, June 2025. https://arxiv.org/abs/2506.07310
9. Edward J. Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models," arXiv:2106.09685, October 2021. https://arxiv.org/abs/2106.09685
10. arXiv, "Non-exclusive License to Distribute," license page. https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html