NVIDIA Cosmos Framework

NVIDIA Cosmos Framework is NVIDIA's Python software for training, adapting, evaluating and serving world models. Its public repository contains distributed training infrastructure, inference programs, checkpoint tools, robot-policy serving and post-training recipes. It belongs to the NVIDIA Cosmos platform; Cosmos 3 is a model family that runs through the framework.^[1]

NVIDIA expanded the framework's model integrations in October 2026. The company identifies GR00T, Alpamayo, Qwen, pi0.5 and other compatible Hugging Face models alongside Cosmos 3. Support depends on the selected model, algorithm and runtime contract: a listed integration does not imply that every recipe works with every checkpoint.^[1][2]

Place in the Cosmos platform

Cosmos separates data preparation, model development and application serving. NVIDIA positions Cosmos Data Factory as the data-curation and synthetic-data layer, Cosmos Framework as the training and adaptation layer, and Cosmos NIM microservices as a deployment route for supported models.^[2] The framework also contains its own inference and policy-server code.^[1]

The Cosmos 3 technical report describes a Mixture-of-Transformers architecture that combines autoregressive reasoning with diffusion-based generation across language, images, video, audio and actions. Those are model capabilities described by the research; which operations a developer can run depends on the released checkpoint and its runtime. The framework's inference documentation, for example, excludes audio-enabled examples for the released Edge checkpoint because it lacks a sound tokenizer.^[3][4]

Main components

The repository uses a top-level cosmos_framework package and separate integration packages for downstream inference ecosystems.^[13]

The table summarizes the repository architecture and its separately documented policy and post-training paths.^[8][10][13]

These components serve different stages of development. Installing the package does not download every model, dataset or dependency required by every workflow.^[1][5]

Supervised adaptation

The native training guide covers fine-tuning Cosmos 3 generators, reasoners and action policies. Its runnable recipes include vision adaptation and robot-policy training. Data preparation, checkpoint conversion, training and checkpoint export are separate steps. Training runs save a configuration that can later supply the dataset settings for evaluation.^[6][7]

Full supervised fine-tuning updates model parameters, while supported LoRA recipes use low-rank adaptation. The reference vision recipes are tested on eight H100 GPUs with 80 GB each. This is a documented recipe configuration, rather than a universal minimum for the whole framework; resource needs depend on the model and parallelism settings.^[6]

The main Cosmos 3 training path and the Hugging Face-based post-training path are distinct. The latter provides supervised adaptation for supported Qwen text and vision-language models outside the native Cosmos recipes. Developers extending it must match the model and data interfaces rather than assume checkpoint compatibility from its file format alone.^[14]

Reinforcement learning and distillation

The cosmos_framework.posttrain runtime supports offline and online post-training from a single host to a distributed Ray cluster. Its public guides distinguish the following methods.^[8]

The guides provide extension points for datasets, rewards, objectives and simulators. They also record small public recipe runs that completed initialization and optimizer updates. Those checks establish that the example pipeline ran; they do not constitute a benchmark showing that every model improves on its target task.^[8]

Inference and policy serving

The inference entry point is python -m cosmos_framework.scripts.inference. It reads sample configurations from JSON, JSONL or YAML files and writes generation settings, completion status and enabled visual or action outputs. A single-GPU invocation and multi-GPU torchrun recipes are documented separately. Model weights, activations, tokenizers and output buffers all contribute to memory requirements.^[4]

Backend choices include Diffusers, Transformers and vLLM, with compatibility depending on the model and task. The November 2026 technical guide also describes optimized deployment options including vLLM-Omni, TensorRT-based runtimes and NIM.^[9] These are alternative supported paths, not a statement that every backend handles every Cosmos modality.

For DROID robot policies, NVIDIA documents separate Nano and Edge checkpoints. A policy server streams predicted actions to a client controlling a simulated or real robot. The reference client uses RoboLab, with examples for an interactive task viewer and parallel headless evaluation.^[10]

Evaluation

The framework's evaluation program can generate predictions from a checkpoint and compare them with ground truth, or score already-generated videos without loading a model. Its documented action modes use peak signal-to-noise ratio for forward-dynamics video, mean squared error for inverse-dynamics actions, or both for world-action modeling. The default dataset split is validation.^[7]

Evaluation writes per-sample results and aggregates means and sample counts. Its video-only path skips clips with incompatible spatial dimensions, trims unequal frame counts to the shorter sequence and skips predictions without matching ground truth. These details affect which samples contribute to a reported result.^[7]

Environment and reproducibility

The setup guide specifies Linux on x86-64 or aarch64, Python 3.10 or newer, glibc 2.35 or newer, CUDA 12.8 or newer and an NVIDIA GPU from Ampere onward. Hopper or Blackwell is recommended for full training throughput. Multinode work also needs a functioning NCCL setup and shared checkpoint storage.^[5]

NVIDIA recommends about 150 GiB of free disk for an initial workflow, including caches and outputs. Its supported installation routes use a CUDA-matched dependency group or a container; the published examples pin a specific uv version. GPU and media-runtime checks precede checkpoint downloads.^[5]

The repository supplies task-specific instructions for coding agents under .agents/skills, with a mirror under .claude/skills. They cover setup, inference, post-training, repository navigation and troubleshooting.^[11] NVIDIA's November guide recommends bounded agent tasks with pinned revisions, separate training and evaluation episodes, a smoke test and a GPU-hour limit. It also recommends recording quality, memory and latency when comparing precision settings.^[9]

License

The repository's root license is OpenMDW-1.1. It grants rights to use, modify and distribute the covered model materials, which its definition includes associated software, data and documentation. Redistribution must retain the agreement and applicable notices. The agreement includes a patent-and-copyright-litigation termination provision and disclaims warranties.^[12]

The root license applies to materials distributed under that agreement. Checkpoint downloads, datasets and external dependencies must be checked against their own applicable notices and terms; a framework installation is not a license grant for every third-party artifact it can load.^[12]

Continue reading

• NvidiaNvidia Corporation is an American technology company that designs graphics processing units (GPUs) and is the world's dominant supplier of hardware for artificial intelligenceMentioned 9 times above
• NNVIDIA Cosmos 3NVIDIA Cosmos 3 is an open family of "world foundation models" for physical AI that NVIDIA launched on June 1, 2026 at GTC Taipei, held alongside COMPUTEX 2026.Mentioned 5 times above
• NVIDIA CosmosNVIDIA Cosmos is a world foundation model platform developed by NVIDIA for physical AI applications, including autonomous vehicles and robotics.Mentioned twice above
• WWorld modelA world model is an artificial intelligence system that learns an internal representation of how an environment works, enabling it to predict future states, simulate the consequences of actions, and support…Linked in the introduction
• WWorld action modelA world action model (WAM) is a robot policy design that builds action generation on a video world model backbone rather than on a vision-language model, so that a single network jointly predicts how a scene…Also in World Models
• V-JEPA 2V-JEPA 2 (Video Joint Embedding Predictive Architecture 2) is an open-source video world model released by Meta AI on June 11, 2025 that learns to understand, predict, and plan in the physical world by…Also in World Models

More to read

• Jeff Hawke
• Odyssey-3
• Dream-1
• Mimic Robotics
• AR-WAM
• Lumo-2

Topics:RoboticsSoftware DevelopmentWorld Models

References

1. ^1 ^2 ^3 ^4NVIDIA. Cosmos-Framework repository and README. Retrieved November 4, 2026.
2. ^1 ^2Pranjali Joshi. NVIDIA Cosmos Opens New Tools to Train, Evaluate and Deploy World Models. NVIDIA Blog, October 21, 2026.
3. ^NVIDIA et al. Cosmos 3: Omnimodal World Models for Physical AI. arXiv:2606.02800, submitted June 1, 2026.
4. ^1 ^2NVIDIA. Cosmos Framework inference guide. Retrieved November 4, 2026.
5. ^1 ^2 ^3NVIDIA. Cosmos Framework setup guide. Retrieved November 4, 2026.
6. ^1 ^2NVIDIA. Post-Training: Supervised Fine-Tuning. Retrieved November 4, 2026.
7. ^1 ^2 ^3NVIDIA. Cosmos Framework evaluation guide. Retrieved November 4, 2026.
8. ^1 ^2 ^3NVIDIA. Cosmos Framework post-training documentation. Retrieved November 4, 2026.
9. ^1 ^2Pranjali Joshi et al. Build World Models for Interactive Simulation with NVIDIA Cosmos. NVIDIA Technical Blog, November 2, 2026.
10. ^1 ^2NVIDIA. Cosmos3-Policy-DROID Server. Retrieved November 4, 2026.
11. ^NVIDIA. Cosmos Framework agent skills. Retrieved November 4, 2026.
12. ^1 ^2NVIDIA. OpenMDW License Agreement, version 1.1. Retrieved November 4, 2026.
13. ^1 ^2NVIDIA. Cosmos Framework code structure. Retrieved November 4, 2026.
14. ^NVIDIA. Hugging Face-based supervised fine-tuning guide. Retrieved November 4, 2026.

Show all 14 references