# Lyra 2.0

> Source: https://aiwiki.ai/wiki/lyra_2_0
> Updated: 2026-11-08
> Categories: AI Research, Computer Vision, NVIDIA, World Models
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Lyra 2.0." aiwiki.ai, 8 Nov 2026. https://aiwiki.ai/wiki/lyra_2_0
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Lyra 2.0** is a research framework from [NVIDIA](https://aiwiki.ai/wiki/nvidia) that turns a single image and a camera path into an explorable 3D scene. It first generates a camera-controlled video, then reconstructs that video as [3D Gaussian splats](https://aiwiki.ai/wiki/gaussian_splatting) and, in the research pipeline, a surface mesh. Its main research problem is keeping a generated environment visually consistent while the camera travels farther and returns to places seen earlier.[1][2]

The paper, *Lyra 2.0: Explorable Generative 3D Worlds*, was submitted to arXiv on April 14, 2026. The project announced public weights and inference code on April 15 and later released training code and an interactive viewer. The project page lists acceptance to SIGGRAPH Asia 2026. The source code has an Apache 2.0 license, while the model weights have a separate license restricted to internal scientific research and development.[1][3][4][5]

## Overview

| Property | Description |
|---|---|
| Developer | NVIDIA Spatial Intelligence Lab |
| Research task | Single-image generation of explorable, predominantly static 3D environments |
| Backbone | [Wan 2.1](https://aiwiki.ai/wiki/wan_2_1) 14B video diffusion transformer |
| Generation inputs | Starting image, camera trajectory and optional text conditioning |
| Generation resolution | 832 x 480 pixels |
| Intermediate output | Video of the requested camera movement |
| Reconstruction output | 3D Gaussian scene; the paper also describes surface mesh extraction |
| Official implementation | `nv-tlabs/lyra`, under `Lyra-2` |
| Official weights | `nvidia/Lyra-2.0` on Hugging Face |
| Code license | Apache 2.0 |
| Model license | NVIDIA Internal Scientific Research and Development Model License |

The distinction between generation and rendering matters: an exported Gaussian scene can be rendered interactively after reconstruction, but generating new views of unexplored regions requires running the video model. The repository reports generation times measured in seconds or minutes per chunk, rather than video-rate generation on the documented single-GPU configuration.[2][4][5]

## How it works

### Spatial memory and revisiting a scene

A video model has a limited context window. When the camera returns to an area that has fallen outside that window, the model can generate a different layout or change objects that were already visible. Lyra 2.0 stores geometry separately for each historical frame and retrieves views relevant to the next camera position.[2]

The cache contains estimated depth, camera parameters and a downsampled point cloud for each frame. It does not merge all of those point clouds into one global reconstruction during generation. This avoids forcing errors from different depth estimates into a single accumulated geometry model.[2]

For a requested viewpoint, the retrieval procedure scores how much of each historical frame would be visible, accounting for occlusion. The paper's configuration retrieves up to five frames while favoring coverage of areas not already covered by another retrieved view. Those images supply visual context. Warped coordinate and depth maps provide correspondence information that tells the transformer which locations refer to the same parts of the scene.[2]

This design uses estimated geometry to route information between views. Appearance still comes from the generative model, rather than directly copying a rendered point cloud into the output. It improves consistency in the reported experiments, but does not make the generated geometry an exact reconstruction of a real place.[2]

### Reducing drift during long videos

Autoregressive generation conditions later chunks on earlier generated chunks. Small errors in color, texture or shape can therefore accumulate. Lyra 2.0 combines compressed temporal history with self-augmented training histories: during training, the model sometimes receives an imperfect history produced through its own denoising process, while the target remains clean. This trains it to recover from degraded conditioning.[2]

The paper uses FramePack-style compression for temporal history. Its implementation also includes camera-conditioning modules and spatial-memory correspondence embeddings in the video transformer. These mechanisms address different problems: temporal history connects neighboring chunks, while retrieved spatial history helps when a camera revisits an older location.[2]

### Reconstructing a 3D scene

After generation, a reconstruction stage converts video frames into an explicit scene. The research implementation uses a modified Depth Anything 3 Gaussian-prediction head. The modification reduces the number of predicted Gaussians, and fine-tuning on generated videos helps the reconstruction model handle artifacts in those videos.[2]

The paper also describes extracting meshes from Gaussian-derived depth and normals using a hierarchical sparse grid. Its demonstrations import reconstructed scenes into [NVIDIA Isaac Sim](https://aiwiki.ai/wiki/nvidia_isaac_sim) for robot navigation and interaction. These are examples of using generated assets in a simulator; they do not establish the accuracy of a real building's dimensions, materials or physical behavior.[1][2]

## Documented workflows

The official repository provides three ways to control the generated camera movement.[5]

| Workflow | Inputs and behavior |
|---|---|
| Preset trajectory | A starting image and caption, with adjustable zoom-in and zoom-out motion |
| Custom trajectory | A starting image, camera poses and intrinsics in a trajectory file, plus optional captions for individual chunks |
| Interactive viewer | A 3D viewport for authoring camera paths, extending the scene and inspecting accumulated point clouds |

The custom-trajectory format includes world-to-camera matrices and camera intrinsics. Captions can change between chunks, allowing text conditioning to describe newly revealed regions. The documented frame counts align with 80-frame autoregressive chunks, using counts such as 81, 161 or 241.[5]

In the interactive workflow, a local viewer connects to an inference server on the same machine or a remote machine. An image seeds the scene, after which users record or edit a camera path and generate another chunk. A vision-language model rewrites the supplied description using the current scene context; an empty description lets it propose content. A revert control returns to the state before the latest chunk. The stitched video is then available for the separate reconstruction step.[6]

The viewer documentation requires Linux, an NVIDIA GPU and a CUDA toolkit. The main installation guide reports testing on Ubuntu 22.04, CUDA 12.8 and H100 GPUs. It says other Linux distributions and CUDA 12.4 or newer may work, while distinguishing them from the verified setup.[6][7]

### Generation speed and the distilled option

The repository reports the following approximate runtimes on one H100 with 80 GB of memory.[5]

| Stage | Documented runtime |
|---|---:|
| Full video generation | About 9 minutes per 80 frames |
| Video generation with the optional DMD LoRA | About 35 seconds per 80 frames |
| ViPE, Depth Anything 3 and Gaussian rendering reconstruction pipeline | About 1 minute |

These are developer-reported example timings, not a guarantee for every scene or installation. The reconstruction timing is separate from video generation.[5]

The optional Distribution Matching Distillation (DMD) LoRA reduces generation to four denoising steps and removes the need for separate conditional and unconditional guidance passes. The repository warns that this option can weaken prompt following and create repetitive patterns. It recommends the full sampler for the best quality and scene scale.[5]

For videos longer than approximately 500 frames, the repository recommends overlapping reconstruction chunks rather than sending the entire sequence through one Depth Anything 3 pass. The Gaussian head was not fine-tuned at those sequence lengths, and the documentation warns that a single pass can produce blurry results.[5]

## Training and evaluation

The paper trains the video generator using DL3DV, a dataset of real-world scene videos, with estimated camera poses, depth and generated captions. The released training guide describes preprocessing long static-scene videos, keeping camera motion and other modalities aligned, dividing videos into 80-frame chunks and encoding their captions. It includes sample training data and a short test run for checking the data and visualization pipeline.[2][8]

### Long-video benchmarks

The authors evaluate camera-controlled video generation on DL3DV and Tanks and Temples. They report image similarity and perceptual metrics, style consistency, camera controllability and reprojection error. Selected results from the paper's Table 1 are shown below. SSIM and camera-control scores are higher-is-better; LPIPS, FID and reprojection error are lower-is-better.[2]

| Dataset and method | SSIM | LPIPS | FID | Camera control | Reprojection error |
|---|---:|---:|---:|---:|---:|
| DL3DV, Lyra 2.0 full | 0.388 | 0.498 | 43.43 | 64.67 | 0.076 |
| DL3DV, Lyra 2.0 DMD | 0.359 | 0.507 | 43.63 | 65.64 | 0.088 |
| DL3DV, GEN3C | 0.346 | 0.535 | 58.96 | 69.54 | 0.068 |
| Tanks and Temples, Lyra 2.0 full | 0.384 | 0.552 | 51.33 | 63.87 | 0.069 |
| Tanks and Temples, Lyra 2.0 DMD | 0.362 | 0.545 | 49.71 | 58.12 | 0.077 |
| Tanks and Temples, GEN3C | 0.350 | 0.589 | 79.07 | 70.91 | 0.054 |

The results show a tradeoff rather than a clean win on every measure. In these comparisons, the full Lyra model has better image-quality measures than GEN3C, while GEN3C has stronger camera-control and reprojection results. The distilled model changes different metrics in different directions.[2]

For reconstructed 3D scenes, the paper compares generated videos followed by the same pretrained reconstruction model, then evaluates a version with a reconstruction model fine-tuned on generated scenes. On Tanks and Temples, the authors report FID of 79.36 for Lyra video with pretrained Depth Anything 3 and 72.47 for the full pipeline, with LPIPS against ground-truth views changing from 0.648 to 0.629. These are results from the authors' benchmark setup, rather than an independent reproduction or a general measure of physical accuracy.[2]

## Limitations

Lyra 2.0 focuses on static environments and does not explicitly model dynamic scenes. The paper also notes that exposure changes in DL3DV can carry into generated videos and introduce artifacts in reconstruction.[2]

Camera trajectories depend on depth estimated from a single starting image. The repository warns that a predefined path can encounter hallucinated objects or produce degenerate outputs, and that its apparent scale can differ from the intended movement. Adjusting trajectory scale is part of the documented workflow.[5]

A generated scene is therefore an inferred environment, including content outside the original image. Visual plausibility, consistency on a camera path, reconstruction quality and correct real-world physics are separate properties. The project's simulator demonstrations support experimentation with generated environments, while its reported benchmarks primarily assess visual and geometric outputs.[1][2]

## Availability and licensing

Official code and weights are available through the project's GitHub and Hugging Face repositories. The public training documentation and viewer instructions support running and modifying the research pipeline.[4][6][8]

Code availability and model-use rights differ. The repository licenses source code under Apache 2.0. The model card instead names the NVIDIA Internal Scientific Research and Development Model License, which permits internal scientific research and development in a non-production environment and restricts distribution, deployment and production use of the model or derivative models. It also restricts using them to generate works for sale or distribution. The repository provides a separate route for requesting a custom license.[4][5][9]

## References

1. NVIDIA Spatial Intelligence Lab. [Lyra 2.0: Explorable Generative 3D Worlds, official project page](https://nv-tlabs.github.io/Project-Lyra/). Release timeline, demonstrations and publication status. Accessed November 8, 2026.
2. Tianchang Shen et al. [Lyra 2.0: Explorable Generative 3D Worlds](https://arxiv.org/html/2604.13036v1). arXiv:2604.13036v1, April 14, 2026. Methods, experiments, limitations and appendix.
3. arXiv. [Lyra 2.0 submission record](https://arxiv.org/abs/2604.13036). Submitted April 14, 2026.
4. NVIDIA. [Lyra-2.0 model card](https://huggingface.co/nvidia/Lyra-2.0). Model architecture, official weights and licensing. Accessed November 8, 2026.
5. NVIDIA. [Lyra 2.0 implementation README](https://github.com/nv-tlabs/lyra/tree/main/Lyra-2). Inference, DMD option, reconstruction, runtimes and license distinction. Accessed November 8, 2026.
6. NVIDIA. [Lyra 2.0 interactive GUI instructions](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/README.md). Client/server arrangement and scene-editing workflow. Accessed November 8, 2026.
7. NVIDIA. [Lyra 2.0 installation guide](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/INSTALL.md). Tested environment and dependencies. Accessed November 8, 2026.
8. NVIDIA. [Training Lyra 2.0](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/TRAINING.md). Training initialization, dataset preprocessing and examples. Accessed November 8, 2026.
9. NVIDIA. [NVIDIA Internal Scientific Research and Development Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-internal-scientific-research-and-development-model-license/). Last modified April 11, 2025.