# NVIDIA Cosmos Framework

> Source: https://aiwiki.ai/wiki/nvidia_cosmos_framework
> Updated: 2026-11-04
> Categories: Robotics, Software Development, World Models
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "NVIDIA Cosmos Framework." aiwiki.ai, 4 Nov 2026. https://aiwiki.ai/wiki/nvidia_cosmos_framework
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**NVIDIA Cosmos Framework** is [NVIDIA's](https://aiwiki.ai/wiki/nvidia) Python software for training, adapting, evaluating and serving [world models](https://aiwiki.ai/wiki/world_model). Its public repository contains distributed training infrastructure, inference programs, checkpoint tools, robot-policy serving and post-training recipes. It belongs to the [NVIDIA Cosmos](https://aiwiki.ai/wiki/nvidia_cosmos) platform; [Cosmos 3](https://aiwiki.ai/wiki/nvidia_cosmos_3) is a model family that runs through the framework.[1]

NVIDIA expanded the framework's model integrations in October 2026. The company identifies GR00T, Alpamayo, Qwen, pi0.5 and other compatible Hugging Face models alongside Cosmos 3. Support depends on the selected model, algorithm and runtime contract: a listed integration does not imply that every recipe works with every checkpoint.[1][2]

## Place in the Cosmos platform

Cosmos separates data preparation, model development and application serving. NVIDIA positions Cosmos Data Factory as the data-curation and synthetic-data layer, Cosmos Framework as the training and adaptation layer, and Cosmos NIM microservices as a deployment route for supported models.[2] The framework also contains its own inference and policy-server code.[1]

The Cosmos 3 technical report describes a Mixture-of-Transformers architecture that combines autoregressive reasoning with diffusion-based generation across language, images, video, audio and actions. Those are model capabilities described by the research; which operations a developer can run depends on the released checkpoint and its runtime. The framework's inference documentation, for example, excludes audio-enabled examples for the released Edge checkpoint because it lacks a sound tokenizer.[3][4]

## Main components

The repository uses a top-level `cosmos_framework` package and separate integration packages for downstream inference ecosystems.[13]

| Component | Documented role |
| --- | --- |
| Training | Distributed training with fully sharded data, tensor, context and pipeline parallelism |
| Checkpoints | Native distributed checkpoints and import or export of Hugging Face `safetensors` |
| Data adapters | JSONL, WebDataset, LeRobot and custom data inputs |
| Inference | Offline generation and online serving through supported backends |
| Post-training | Supervised, preference-based, reinforcement-learning and distillation workflows |
| Robot-policy server | A server-client connection that supplies action predictions to a robot or simulator |

The table summarizes the repository architecture and its separately documented policy and post-training paths.[8][10][13]

These components serve different stages of development. Installing the package does not download every model, dataset or dependency required by every workflow.[1][5]

## Supervised adaptation

The native training guide covers fine-tuning Cosmos 3 generators, reasoners and action policies. Its runnable recipes include vision adaptation and robot-policy training. Data preparation, checkpoint conversion, training and checkpoint export are separate steps. Training runs save a configuration that can later supply the dataset settings for evaluation.[6][7]

Full supervised fine-tuning updates model parameters, while supported LoRA recipes use [low-rank adaptation](https://aiwiki.ai/wiki/low_rank_adaptation). The reference vision recipes are tested on eight H100 GPUs with 80 GB each. This is a documented recipe configuration, rather than a universal minimum for the whole framework; resource needs depend on the model and parallelism settings.[6]

The main Cosmos 3 training path and the Hugging Face-based post-training path are distinct. The latter provides supervised adaptation for supported Qwen text and vision-language models outside the native Cosmos recipes. Developers extending it must match the model and data interfaces rather than assume checkpoint compatibility from its file format alone.[14]

## Reinforcement learning and distillation

The `cosmos_framework.posttrain` runtime supports offline and online post-training from a single host to a distributed [Ray](https://aiwiki.ai/wiki/ray) cluster. Its public guides distinguish the following methods.[8]

| Method | Purpose in the documented runtime |
| --- | --- |
| [Direct preference optimization](https://aiwiki.ai/wiki/direct_preference_optimization) | Offline learning from preference pairs |
| GRPO | Online optimization with rule-based or model-based rewards and vLLM rollouts |
| [Proximal policy optimization](https://aiwiki.ai/wiki/proximal_policy_optimization) | Actor-critic optimization with reference and critic roles |
| On-policy distillation | Training a policy from a frozen teacher on the policy's own generated samples |
| DiffusionNFT | Online reinforcement learning for diffusion generators |
| Joint embodied PPO | Policy learning with simulator-based interaction through supported integrations |

The guides provide extension points for datasets, rewards, objectives and simulators. They also record small public recipe runs that completed initialization and optimizer updates. Those checks establish that the example pipeline ran; they do not constitute a benchmark showing that every model improves on its target task.[8]

## Inference and policy serving

The inference entry point is `python -m cosmos_framework.scripts.inference`. It reads sample configurations from JSON, JSONL or YAML files and writes generation settings, completion status and enabled visual or action outputs. A single-GPU invocation and multi-GPU `torchrun` recipes are documented separately. Model weights, activations, tokenizers and output buffers all contribute to memory requirements.[4]

Backend choices include Diffusers, Transformers and [vLLM](https://aiwiki.ai/wiki/vllm), with compatibility depending on the model and task. The November 2026 technical guide also describes optimized deployment options including vLLM-Omni, TensorRT-based runtimes and NIM.[9] These are alternative supported paths, not a statement that every backend handles every Cosmos modality.

For DROID robot policies, NVIDIA documents separate Nano and Edge checkpoints. A policy server streams predicted actions to a client controlling a simulated or real robot. The reference client uses RoboLab, with examples for an interactive task viewer and parallel headless evaluation.[10]

## Evaluation

The framework's evaluation program can generate predictions from a checkpoint and compare them with ground truth, or score already-generated videos without loading a model. Its documented action modes use peak signal-to-noise ratio for forward-dynamics video, mean squared error for inverse-dynamics actions, or both for world-action modeling. The default dataset split is validation.[7]

Evaluation writes per-sample results and aggregates means and sample counts. Its video-only path skips clips with incompatible spatial dimensions, trims unequal frame counts to the shorter sequence and skips predictions without matching ground truth. These details affect which samples contribute to a reported result.[7]

## Environment and reproducibility

The setup guide specifies Linux on x86-64 or aarch64, Python 3.10 or newer, glibc 2.35 or newer, CUDA 12.8 or newer and an NVIDIA GPU from Ampere onward. Hopper or Blackwell is recommended for full training throughput. Multinode work also needs a functioning NCCL setup and shared checkpoint storage.[5]

NVIDIA recommends about 150 GiB of free disk for an initial workflow, including caches and outputs. Its supported installation routes use a CUDA-matched dependency group or a container; the published examples pin a specific `uv` version. GPU and media-runtime checks precede checkpoint downloads.[5]

The repository supplies task-specific instructions for coding agents under `.agents/skills`, with a mirror under `.claude/skills`. They cover setup, inference, post-training, repository navigation and troubleshooting.[11] NVIDIA's November guide recommends bounded agent tasks with pinned revisions, separate training and evaluation episodes, a smoke test and a GPU-hour limit. It also recommends recording quality, memory and latency when comparing precision settings.[9]

## License

The repository's root license is **OpenMDW-1.1**. It grants rights to use, modify and distribute the covered model materials, which its definition includes associated software, data and documentation. Redistribution must retain the agreement and applicable notices. The agreement includes a patent-and-copyright-litigation termination provision and disclaims warranties.[12]

The root license applies to materials distributed under that agreement. Checkpoint downloads, datasets and external dependencies must be checked against their own applicable notices and terms; a framework installation is not a license grant for every third-party artifact it can load.[12]

## References

1. NVIDIA. [Cosmos-Framework repository and README](https://github.com/NVIDIA/cosmos-framework). Retrieved November 4, 2026.
2. Pranjali Joshi. [NVIDIA Cosmos Opens New Tools to Train, Evaluate and Deploy World Models](https://blogs.nvidia.com/blog/cosmos-open-train-evaluate-deploy-world-models/). NVIDIA Blog, October 21, 2026.
3. NVIDIA et al. [Cosmos 3: Omnimodal World Models for Physical AI](https://arxiv.org/abs/2606.02800). arXiv:2606.02800, submitted June 1, 2026.
4. NVIDIA. [Cosmos Framework inference guide](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/inference.md). Retrieved November 4, 2026.
5. NVIDIA. [Cosmos Framework setup guide](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/setup.md). Retrieved November 4, 2026.
6. NVIDIA. [Post-Training: Supervised Fine-Tuning](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/training.md). Retrieved November 4, 2026.
7. NVIDIA. [Cosmos Framework evaluation guide](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/evaluation.md). Retrieved November 4, 2026.
8. NVIDIA. [Cosmos Framework post-training documentation](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/posttrain/index.md). Retrieved November 4, 2026.
9. Pranjali Joshi et al. [Build World Models for Interactive Simulation with NVIDIA Cosmos](https://developer.nvidia.com/blog/build-world-models-for-interactive-simulation-with-nvidia-cosmos). NVIDIA Technical Blog, November 2, 2026.
10. NVIDIA. [Cosmos3-Policy-DROID Server](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/action_policy_droid_server.md). Retrieved November 4, 2026.
11. NVIDIA. [Cosmos Framework agent skills](https://nvidia.github.io/cosmos-framework/get_started/agent_skills/). Retrieved November 4, 2026.
12. NVIDIA. [OpenMDW License Agreement, version 1.1](https://github.com/NVIDIA/cosmos-framework/blob/main/LICENSE). Retrieved November 4, 2026.

13. NVIDIA. [Cosmos Framework code structure](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/code_structure.md). Retrieved November 4, 2026.
14. NVIDIA. [Hugging Face-based supervised fine-tuning guide](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/posttrain/guides/sft.md). Retrieved November 4, 2026.