MotionBricks

MotionBricks is a real-time generative model for character animation and humanoid robot motion developed by researchers at NVIDIA, with co-authors who also hold appointments at ETH Zurich, Simon Fraser University and the University of Texas at Austin. Described in the paper "MotionBricks: Scalable Real-Time Motions with Modular Latent Generative Model and Smart Primitives," published in ACM Transactions on Graphics and presented at SIGGRAPH 2026, it pairs a single latent generative "backbone" that fills in motion between keyframes with a high-level authoring layer called smart primitives.^[1][2] The authors report that one model covers a proprietary dataset of about 350,000 motion clips while generating motion at 15,000 frames per second with 2 milliseconds of latency, and that the same architecture and training setup drives both an Unreal Engine 5 game demo and, trained on data retargeted to the robot's skeleton, a Unitree G1 humanoid robot.^[1] NVIDIA first released code as a preview inside its GR00T Whole-Body Control repository on 27 April 2026 and moved it to a standalone repository, with an added pretrained model for the SOMA human skeleton, in October 2026.^[6][12]

Background

Interactive games and many robot behavior systems still rely on animation graphs (also called behavior graphs or motion graphs): state machines in which recorded clips such as walking, running or idling are organized into states, with transitions triggered by player input or game events.^[1] The MotionBricks paper argues that this approach scales poorly, citing a 2018 Game Developers Conference talk by Daniel Holden for the figure that a modern AAA animation state machine such as the one in Assassin's Creed can involve more than 15,000 animations, 5,000 states and graphs nested up to 12 levels deep.^[1] Motion matching, which retrieves future frames from a motion database to fit the current state and control input, removed some of the rigidity of node-based graphs but, according to the authors, is usually still embedded inside a larger graph as a navigation component.^[1]

On the research side, the paper divides earlier learned approaches into three groups. Deterministic neural controllers, of which the paper calls Phase-Functioned Neural Networks (Holden et al., 2017) a seminal example, predict motion autoregressively but tend to be specialized and show artifacts such as foot sliding and over-smoothing. Motion in-betweening methods fill in frames between given start and end keyframes but are mostly used offline because target keyframes are not normally available at runtime. Generative models, including motion diffusion models and token-based models built on vector quantization, scale better but, in the authors' assessment, are either too slow for real-time use or weak at satisfying precise spatial constraints.^[1] The paper frames two gaps between research and production: generative methods degrade in quality and scalability when asked to produce a very large repertoire of skills under real-time compute budgets, and existing text- or tag-driven models largely do not offer fine-grained control such as velocity commands, style selection and exact keyframes.^[1]

Architecture

MotionBricks treats every task as motion in-betweening. "Context" keyframes describe the character's most recent poses, and "target" keyframes, supplied by smart primitives, describe where the motion should go. Each module accepts an arbitrary subset of constraints on the local root, global root and pose, and substitutes a learned mask embedding for any constraint that is missing, so the same network can handle sparse navigation targets and dense object-interaction targets. In-between segments range from 12 to 64 frames at 30 frames per second.^[1] Inference runs in four stages: smart primitives build keyframes, a root module predicts timing and root trajectory, a pose module predicts latent pose tokens, and a decoder produces continuous motion. The system runs autoregressively and replans when control signals change or when the buffer of future frames runs low.^[1]

^[1]

Multi-head tokenizer

Instead of a single large codebook or a manual split by body part, the tokenizer quantizes the latent with several independent codebooks ("heads") and lets the network learn how to divide the representation. The authors argue this captures more of the variety in human movement and degrades more gracefully when individual tokens are mispredicted.^[1] It is trained as a VQ-VAE with a running-mean codebook update; the paper reports that finite scalar quantization (FSQ) performs similarly but that VQ-VAE gave slightly lower cross-entropy loss and better FID, so VQ-VAE is the default. Root and pose are encoded separately, an idea the paper attributes to foot-scaling and time-warping in traditional animation, which lets the decoder warp the same pose tokens onto different root trajectories without introducing foot skating.^[1] In the ablations, the authors recommend 128 to 256 tokens per head and report that a single-head VQ-VAE baseline plateaued as its codebook grew, while the multi-head tokenizer kept improving.^[1]

Root and pose modules

The root module first outputs a distribution over in-between frame counts, which can be sampled, taken as the argmax, or masked to force a specific duration. Its second stage then predicts the initial root path. The pose module follows the masked token modeling approach used in earlier image and motion generators, which predicts tokens iteratively by confidence. The paper notes that at inference a single forward pass usually gives high-quality motion, with the cosine masking schedule used mainly as a training curriculum.^[1] During training, the root module, pose module and decoder each see a random number of keyframe constraints (0 to 10) at arbitrary positions, which the authors say encourages generalizable and robust features. Because every smart primitive communicates through the same keyframe interface, the paper says the trained backbone needs no task-specific tags or fine-tuning for new uses.^[1]

Smart primitives

Smart primitives are the authoring layer. They turn user commands and game events into target keyframes for the backbone, so the paper describes the result as "essentially creating a fully connected motion graph where transitions are generated with the neural backbone."^[1] There are two kinds.

Smart locomotion handles navigation. It extrapolates the commanded velocity and heading over a 1.0 second horizon, smooths that path with a critically damped spring (a technique borrowed from motion matching), and places style keyframes taken from short reference clips or authored poses onto it. The root module then refines the trajectory using predicted timing, which the authors say produces style-specific paths and avoids invalid or abrupt velocities when a user requests an infeasible command (a "control dead zone"). The decoder makes a final adjustment that coordinates the root with footsteps.^[1] The project page shows zombie, injured-leg, injured-torso, skipping, strafing and crouch-strafing styles, plus continuous transitions between idle, walk, jog and run.^[2] Keyframes can also be produced by blending poses through a 2D blendspace, a standard game-engine technique.^[1]

Smart objects handle interaction with the scene. Each object is defined by sets of intent keyframes (for example, a hand-contact pose and an exit pose for climbing a ledge) and an interaction binding that manages detection volumes, sockets for carried objects, and anchoring of keyframes to the object's transform. A per-set "drop-frame" attribute sets whether a keyframe is a hard constraint that must be reached exactly or soft guidance the motion may leave early. Because keyframes are anchored to the object and rotated at runtime, the same definition works when the character approaches from different angles.^[1] Demonstrated interactions include picking up a sword, falling from different heights, jumping over a bench, vaulting, ledge climbing and sitting.^[1][2] The paper reports that authoring either a smart navigation or a smart object setup took non-expert users less than 10 minutes.^[1]

Training and data

The paper's main dataset is a proprietary motion capture collection of about 700 hours and 350,000 clips, which it describes as covering about 9,300 unique skills in 36 categories, including locomotion, combat, sports and object interaction.^[1] For comparison and scaling studies the authors also used a 140-hour subset of that collection (called "70k" in the paper, with about 62,000 training clips), the public HumanML3D text-to-motion dataset, and LaFAN1-G1, a version of the LaFAN1 dataset retargeted to the Unitree G1 skeleton. All datasets were processed at 30 frames per second and all baselines were retrained on them.^[1]

An open subset of the data was released by Bones Studio as BONES-SEED. The paper describes it as about 140,000 clips, roughly 40 percent of the full collection.^[1] The dataset's Hugging Face card lists 142,220 annotated motions (71,132 original plus 71,088 mirrored), about 288 hours, in SOMA and Unitree G1 formats; access is gated behind a license agreement.^[9] NVIDIA's project page and the GR00T subproject README describe BONES-SEED as "350k production-grade mocap clips," which does not match the figures in the paper or on the dataset card.^[2][7][9]

The paper's models were trained on 32 GPUs (4 nodes of 8) for 2 million updates at a batch size of 256 per GPU. Training took about 7 days for the tokenizer, 3 days for the root module and 7 days for the pose module on NVIDIA H100 GPUs; the authors say 16 GPUs gave similar visual quality and L40S GPUs were about twice as slow.^[1]

Reported results

The authors compared MotionBricks against six in-betweening methods, retraining each on the same data and motion representation: Conditional Motion In-betweening (Kim et al., 2022), the Two-stage Transformer (Qin et al., 2022), the Deep Delta-Interpolator (Oreshkin et al., 2024), CondMDI (Cohan et al., 2024), MMM (Pinyoanuntapong et al., 2024) and CLoSD's diffusion planner (Tevet et al., 2025). The first three are deterministic transformer models, CondMDI and CLoSD are diffusion models, and MMM is a token-based model with a single-head tokenizer.^[1] Metrics covered speed, distribution distance (MMD and FID in a learned motion latent space), a human study, diversity, smoothness, physical plausibility and keyframe precision. The human study had 40 participants, who each viewed outputs from three randomly selected methods and chose the best.^[1]

Selected results on the 350k dataset, as reported by the authors (Table 3 of the paper; lower is better for latency, FID and target keyframe error):

^[1]

Reaching success counts generations that reach the target root position within 5 cm and the target heading within 15 degrees.^[1] The highest-throughput method in the table is the deterministic Conditional In-betweening baseline (27,000 FPS), while MotionBricks has the lowest latency; the paper describes MotionBricks as achieving "one of the best" runtime efficiencies. The authors summarize the pattern as non-generative methods reaching reasonable precision with poor FID and human ratings, and diffusion methods improving distribution quality while struggling with spatial constraints.^[1] On the smaller HumanML3D and LaFAN1-G1 benchmarks, MotionBricks again had the lowest FID and highest reaching success in the paper's Table 4, though the authors note that baselines matched or slightly exceeded it on diversity for those smaller datasets.^[1] MotionBricks' own 2 ms and 15,000 FPS figures were measured on a desktop with an RTX 5090 GPU; for the baselines, the paper took latency from the original papers where reported and otherwise estimated it with modified open-source code.^[1] All of these results are self-reported by the authors.

Applications

Unreal Engine 5 demo

For game animation, the authors implemented the backbone and smart primitives as an Unreal Engine 5 plugin, with models exported to ONNX and run through TensorRT in native C++. Motions are retargeted on the fly to the demo's "Messenger" and "Guard" characters with the engine's built-in retargeter.^[1] Character behavior is governed by a StateTree with three states (locomotion, target pose sequence for object interactions, and falling), and the animation graph invokes the model only when new target keyframes arrive.^[1] The project page hosts a 2 minute 40 second uncut demo and states that every motion in it is generated by neural networks, "no foot-locking, no blending, no collision detection, no hand-authored transitions."^[2]

Humanoid robot control

For robotics, the paper retargets the proprietary dataset to the Unitree G1 skeleton and uses MotionBricks as a kinematic planner whose output is tracked by a separate physics-based controller. In the paper's experiments the tracker is the one from SONIC (Luo et al., 2025), a whole-body motion-tracking controller from an overlapping NVIDIA team.^[1] On the robot, models run on an NVIDIA Jetson Orin through TensorRT, where latency rises to 5 ms per inference; replanning happens at 10 Hz or whenever commands change, which the authors say avoids competing for compute with the latency-sensitive low-level control policy.^[1] The released G1 demo in the GR00T repository lets a user steer a simulated G1 in MuJoCo with the keyboard and switch among styles such as stealth walk, zombie walk, hand crawling and elbow crawling.^[7]

Releases and licensing

The GR00T-WholeBodyControl repository lists MotionBricks alongside the decoupled whole-body controller used in Isaac GR00T N1.5 and N1.6 and the GEAR-SONIC controllers, and the project page describes MotionBricks as part of NVIDIA's GR00T Whole-Body Control effort.^[2][6]

The standalone release includes a Python runtime built on NVIDIA Warp, Newton and TensorRT; two demos (an Inbetweener and a Scene Preview with SOMA and G1 "gym" scenes); a Smart Primitive Editor and Scene Editor; and a standalone ONNX exporter.^[3] Two pretrained models are provided, both trained on motion the repository identifies as Bones RigPlay 1 (RP1):^[3][4][5]

Both models take four context poses and up to four target poses at 30 Hz and output between 24 and 64 raw frames per generation.^[4][5] NVIDIA's documentation reports TensorRT GPU times on a GeForce RTX 4090 of 2.50 ms for the SOMA FP32 model and 1.28 ms for its mixed FP16 variant at a batch size of one, excluding input preparation and output decoding.^[13] Both model cards state that generated motion is not validated for physical robot control and that physical execution requires a separate controller.^[4][5]

The code is licensed under Apache 2.0, the model weights under the NVIDIA Open Model Agreement (which, according to the repository, permits commercial use and redistribution of modified models subject to its terms), and the bundled sample motions under an evaluation-only NVIDIA license.^[3] The BONES-SEED dataset is distributed by Bones Studio under its own license.^[9]

Limitations

The paper lists several limitations. The dataset has thin coverage of rare motions (for example, only one clip of vaulting at 0.5 m), and object interactions are not explicitly modeled. In real-world robot deployment, the smart object primitive is disabled because the robot lacks the ground-truth object poses and terrain geometry available in simulation. As a kinematic planner, MotionBricks can produce physically implausible motions such as self-collisions, or motions beyond hardware limits. Retargeting remains a bottleneck: runtime retargeting in the game demo is fast but low quality, while the offline retargeting used for the G1 took months of iteration.^[1] The released SOMA model card adds that the model has no text or scene-geometry input, so the application must place targets and handle obstacles.^[4]

Authors

The paper has 16 authors. Tingwu Wang and Olivier Dionne are joint first authors, and Wang is the corresponding author and project lead. The remaining authors are Michael De Ruyter, David Minor, Davis Rempe, Kaifeng Zhao, Mathis Petrovich, Ye Yuan, Chenran Li, Zhengyi Luo, Brian Robison, Xavier Blackwell, Bernardo Antoniazzi, Xue Bin Peng, Yuke Zhu and Simon Yuen. All are listed with NVIDIA affiliations; Zhao is also affiliated with ETH Zurich, Peng with Simon Fraser University and Zhu with the University of Texas at Austin. Yuen, Peng and Zhu are credited as advisers.^[1] The University of Texas at Austin's Robot Perception and Learning Lab lists the work as an ACM SIGGRAPH conference paper from July 2026.^[11] Independent coverage included a June 2026 article by the Japanese technology site Gigazine.^[10]

See also

• SONIC (Humanoid Control)
• Isaac GR00T
• Unitree G1
• Whole-body control
• Physical AI
• Humanoid robot

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39Tingwu Wang, Olivier Dionne, et al. "MotionBricks: Scalable Real-Time Motions with Modular Latent Generative Model and Smart Primitives." arXiv:2604.24833, 27 April 2026; ACM Transactions on Graphics 45(4), SIGGRAPH 2026, doi:10.1145/3811334. arxiv.org/...2604.24833
2. ^1 ^2 ^3 ^4 ^5 ^6NVIDIA. "MotionBricks" project page. nvlabs.github.io/motionbricks
3. ^1 ^2 ^3 ^4NVlabs. "motionbricks" GitHub repository README. github.com/...motionbricks
4. ^1 ^2 ^3 ^4NVlabs. "MotionBricks SOMA Model Card." github.com/...soma_model.md
5. ^1 ^2 ^3NVlabs. "MotionBricks Unitree G1 Model Card." github.com/...unitree_g1_model.md
6. ^1 ^2 ^3NVlabs. "GR00T-WholeBodyControl" GitHub repository README. github.com/...GR00T-WholeBodyControl
7. ^1 ^2 ^3NVlabs. "MotionBricks" subproject README in GR00T-WholeBodyControl. github.com/...motionbricks
8. ^NVIDIA. "nvidia/MotionBricks" model repository. Hugging Face. huggingface.co/...MotionBricks
9. ^1 ^2 ^3Bones Studio. "BONES-SEED: Skeletal Everyday Embodiment Dataset." Hugging Face. huggingface.co/...seed
10. ^"NVIDIA announces 'MotionBricks,' which generates over 350,000 different game and robot movements in real time." Gigazine, 15 June 2026. gigazine.net/...20260615-nvidia-motionbricks
11. ^Robot Perception and Learning Lab, University of Texas at Austin. "MotionBricks: Scalable Real-Time Motions with Modular Latent Generative Model and Smart Primitives." rpl.cs.utexas.edu/...wang-siggraph26-motionbricks
12. ^1 ^2NVlabs. "motionbricks" CHANGELOG (v0.2.0, 23 October 2026). github.com/...CHANGELOG.md
13. ^NVlabs. "ONNX inference" documentation, Inference performance section. github.com/...onnx_inference.md