Lumera

Lumera, short for Light-aware Unified Engine-native Reconstruction and Assembly, is an experimental computer vision benchmark and reference pipeline for turning one RGB image into an editable 3D scene. Instead of producing only a rendered view or a fused geometric surface, the proposed pipeline represents foreground objects as labeled meshes with transforms and lighting as editable local sources plus a high-dynamic-range environment estimate. The work first appeared as a computer-vision preprint in July 2026.^[1][2]

Lumera is not a released product or a single downloadable model. The name covers a dataset and benchmark called Lumera-2K, two separately trained parsers called Lumera-Box and Lumera-Light, and an assembly pipeline that combines several existing research systems. As of September 3, 2026, the project page marked both its code and Lumera-2K data as "soon." It did not link parser checkpoints or a complete implementation.^[1][2]

Publication and scope

Scroll sideways for more →

The preprint lists Junhao Chen, Xinghao Chen, Henghaofan Zhang, Zihao Qiao, Saining Zhang, Yongzhi Li, Ruqi Huang, Sisi Li, Yimin Sheng, Jianyi Zhu, and Hao Zhao as authors. Their listed affiliations include Tsinghua University, Nankai University, the University of Electronic Science and Technology of China, Sun Yat-sen University, Nanyang Technological University, and Ophilus.AI.^[1]

The paper frames Lumera as a benchmark and reference pipeline for reconstructing explicit scene entities. It does not present Lumera as an application, hosted service, or production-ready asset tool. It also does not evaluate temporal prediction or action-conditioned simulation, so Lumera is not itself a world model in the common dynamical sense. Its reconstructed scene can instead act as structured input for simulation, graphics, or generative AI systems.^[1]

Research problem

Many single-image 3D methods produce geometry, a radiance representation, or a newly generated scene. Those outputs do not necessarily preserve the source image as a collection of separately movable objects and editable lights. Lumera treats the task as structured parsing for a game engine. The target scene contains labeled object instances, rigid transforms, textured meshes, local parametric lights, an HDR environment probe, and metadata that can be imported into an engine.^[1]

This choice separates scene understanding from asset reconstruction. The parsers first locate objects and lights in a shared metric coordinate system. Other models then segment and reconstruct each object, estimate global illumination, and assemble the pieces. The approach makes individual entities editable, but each stage can pass errors to the next. A wrong depth estimate can shift a box, a shifted box can produce a poor image mask, and a missing light cannot be invented by the final bounded editor.^[1]

Reference pipeline

The paper describes the following sequence:^[1]

Scroll sideways for more →

Depth Anything 3 is a feed-forward visual-geometry model that can estimate depth and camera information from one or more views.^[4] In Lumera, its point cloud is an intermediate coordinate bridge rather than the final reconstructed scene. SAM 3D supplies the object-level mesh reconstruction stage,^[5] while IntrinsicHDR estimates information lost when a normal low-dynamic-range image clips bright regions.^[6]

The current pipeline focuses on foreground assets. Walls, floors, ceilings, terrain, and other large shells are handled by a separate process in the authors' description. This division matters because a scene with convincing individual objects can still be incomplete if its room or terrain geometry is missing.^[1]

Lumera-Box and Lumera-Light

Both parsers begin with SpatialLM-1.1-Qwen-0.5B, a vision-language model architecture for structured point-cloud interpretation.^[1][3] Lumera-Box and Lumera-Light are not two heads on one published checkpoint. The authors fine-tune two independent copies with different training streams, schemas, weights, and decoding settings.^[1]

Lumera-Box emits an open-vocabulary label, a three-dimensional center, a yaw angle, and an extent along each axis for every box. The sequence represents an editable layout rather than a pixel mask. At assembly time, each predicted box is projected into the image and used to prompt image segmentation. The resulting crop helps reconstruct the object's texture and shape, while the box remains its identity and rigid placement boundary.^[1]

Lumera-Light emits a seven-value tuple:

Scroll sideways for more →

The phrase "engine-native light" needs qualification. Lumera-2K records richer Unreal Engine properties, including light type, orientation, units, attenuation, cone settings, color, and temperature. The supervised parser evaluated in the paper predicts only position, RGB, and intensity for modeled non-SkyLight sources. SkyLight is handled separately through the environment estimate, and the seven-value output does not recreate every original Unreal Engine light field.^[1]

Lumera-2K

Lumera-2K is derived from Unreal Engine 5 projects that the authors describe as publicly available free assets collected from the internet. The fixed totals reported in the paper are:^[1]

Scroll sideways for more →

The authors use a headless Unreal Engine pipeline to expand the sparse cameras included with source projects. Those projects average 3.0 existing cameras, while the planner considers 16 to 100 views per project. It clusters foreground objects, treats indoor and outdoor layouts differently, rejects weak viewpoints, selects for coverage and view diversity, and filters blank, overexposed, underexposed, or low-content frames before high-resolution export.^[1]

Labels are not taken directly from internal asset identifiers. The pipeline renders isolated objects and combines visual evidence with weak asset-name hints and a contextual scene image for VLM-assisted labeling. A normalization pass suppresses shell and editor-placeholder labels. This produces open-vocabulary descriptions, but it also means the labels reflect an automated annotation procedure described by the authors.^[1]

For lighting, the extracted source data covers PointLight, SpotLight, DirectionalLight, RectLight, and SkyLight components. The supervised light task narrows this information to active non-SkyLight sources and the seven-value output used by Lumera-Light. The paper later identifies SkyLight, near-camera sources, and sources outside the visible set as incomplete areas.^[1]

Bounded Generator and Verifier refinement

After assembly, Lumera uses an iterative editing design adapted from the Vision-as-Inverse-Graphics Agent approach.^[7] One agent proposes an edit and another inspects the rendered result. Lumera divides this process into geometry and lighting stages, each with a strict field-level whitelist.^[1]

During geometry refinement, the system may change object yaw and scale. Object position, identity, cameras, lights, materials, and mesh topology are frozen by default. During lighting refinement, it may adjust lights already present in the initial scene, environment strength, and exposure. It may not add a missing light or change scene geometry.^[1]

The executor scans proposed code, compares scene state before and after an action, applies stage-specific preconditions, and rolls back edits that violate the active scope. The paper's experiments allow up to 12 rounds for geometry and 12 for lighting. These controls limit collateral changes, but they also constrain recovery: if the parser misses an object or light, the editor is not allowed to create it.^[1]

Reported box-parsing results

The box benchmark uses project-level splits with 1,316 validation views and 1,314 test views. Its merged evaluation therefore covers 2,630 views. Before scoring, the authors sanitize outputs by removing invalid labels, non-positive box sizes, repeated identifiers, and identifier fallback cases. The following selected metrics come from that merged benchmark, and higher values are better for every displayed column.^[1]

Scroll sideways for more →

The authors report that Lumera-Box leads these baselines on their main detection, metric-geometry, semantic, and graph-consistency measures. The comparison also shows two weak areas. WildDet3D has higher SRF and anchor recall, indicating stronger relation-oriented anchor coverage on those measures. Lumera-Box's own mAP of 0.1141 and IoU-B of 0.2472 remain far from perfect scores. The paper says strict 3D overlap is not saturated.^[1]

The sanitization step is another boundary on interpretation. It creates a common scoring protocol, but malformed raw outputs are removed before evaluation. The paper identifies constrained decoding or a parser-side repair layer as future work.^[1]

All figures in this section are reported by Lumera's authors on the benchmark they constructed. The preprint and project page do not provide an independent reproduction.^[1][2]

Reported light-parsing results

The main light evaluation uses 575 scenes with at least one labeled light, split into 284 validation scenes and 291 test scenes. It first matches predicted and reference lights by position, then measures color and intensity on matched pairs.^[1]

Scroll sideways for more →

The 0.998 result is scene-level recall: Lumera-Light nearly always predicts that a nonempty scene contains a light. It does not mean that the system finds 99.8 percent of individual lights. At the paper's 0.5-meter threshold, individual-light recall is 0.201 and F1 is 0.209. The underlying counts are 553 true positives, 1,983 false positives, and 2,195 false negatives.^[1]

Matched lights have a reported median position error of 0.261 meters and median color error of 4.59 in Delta E 2000. These conditional matched-pair metrics do not account for every missed or extra light. Intensity remains coarse: the authors interpret the log10 mean absolute error of 0.431 as about a 2.7-fold multiplicative brightness error. F1 rises to 0.456 when the matching radius is relaxed to 2.0 meters, suggesting that rough light neighborhoods are easier than precise source localization.^[1]

Assembly case study

The paper reports one 55-instance indoor refinement case that begins with ground-truth boxes. After up to 12 geometry rounds, an automated VLM score rises from 6.1 to 8.3, while Chamfer distance falls to 71.8 percent of its initial value. These are author-reported results from a favorable case with reference boxes, not a full end-to-end benchmark of raw single-image reconstruction.^[1]

The same section says the loop provides little benefit on outdoor scenes with poor starting boxes. This result supports the narrower description of the Generator and Verifier as a bounded editor over a usable parse. It does not show that iterative agents can recover arbitrary missing or badly placed scene content.^[1]

Limitations

The authors identify several unresolved problems:^[1]

• Strict box overlap remains low, and yaw estimation does not explicitly handle symmetric objects.
• Relation structure is weaker than WildDet3D on SRF and anchor recall.
• Raw structured outputs can be malformed even though the sanitized benchmark removes those cases.
• Individual-light precision and recall are low at the strict localization threshold.
• Light intensity has about a 2.7-fold typical multiplicative error by the paper's interpretation.
• SkyLight and strong near-camera or out-of-view sources are not fully represented by the current supervised target.
• The outdoor split has Chamfer-L2 around 17 meters, showing large metric drift in unbounded scenes.
• Foreground reconstruction and the separate shell process can leave room or terrain structure incomplete.
• The extraction pipeline and adapter are implemented for Unreal Engine 5 data. The paper does not demonstrate training-data generalization across Unity, Godot, or other engines.

These limits affect different parts of the pipeline. A plausible rendered image does not establish correct object identity, metric placement, source lighting, or editability. Likewise, strong performance relative to the selected baselines does not mean that the benchmark is solved.^[1]

Availability and reproducibility

As of September 3, 2026, the public project page showed disabled controls labeled "Code - soon" and "Lumera-2K - soon." The reviewed sources did not link Lumera-Box or Lumera-Light checkpoints, the full assembly code, or a dataset download. They also did not identify software, checkpoint, or dataset licenses.^[1][2]

The arXiv manuscript is distributed under CC BY-NC-SA 4.0.^[1][8] That license covers the paper itself. It does not automatically license the unreleased code, model weights, Lumera-2K files, or the Unreal Engine projects from which the dataset was derived.

This release state prevents independent reproduction from the described Lumera artifacts alone. The underlying components have their own papers and some have separate public implementations, but that does not substitute for the missing Lumera training data, parser weights, preprocessing, and integration code. The reported quantitative and qualitative results should therefore be read as claims from a July 2026 preprint and its author-controlled project page.^[1][2]

References

1. ^Junhao Chen et al., "Engine-Native Editable 3D World Reconstruction with Objects and Lighting," arXiv:2607.20889v1, July 23, 2026. arxiv.org/...2607.20889
2. ^Lumera authors, "Lumera: Engine-Native Editable 3D World Reconstruction," project page, 2026. haidilao0328.github.io/Lumera
3. ^Yongsen Mao et al., "SpatialLM: Training Large Language Models for Structured Indoor Modeling," NeurIPS 2025. proceedings.neurips.cc/...7939-Abstract-Conference
4. ^Haotong Lin et al., "Depth Anything 3: Recovering the Visual Space from Any Views," arXiv:2511.10647, November 2025. arxiv.org/...2511.10647
5. ^SAM 3D Team et al., "SAM 3D: 3Dfy Anything in Images," arXiv:2511.16624, revised June 2026. arxiv.org/...2511.16624
6. ^Sebastian Dille, Chris Careaga, and Yağız Aksoy, "Intrinsic Single-Image HDR Reconstruction," ECCV 2024. arxiv.org/...2409.13803
7. ^Shaofeng Yin et al., "Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning," arXiv:2601.11109v3, April 2026. arxiv.org/...2601.11109
8. ^Creative Commons, "Attribution-NonCommercial-ShareAlike 4.0 International." creativecommons.org/...4.0