Inverse graphics

Inverse graphics is the problem of recovering, from one or more images, a description of the scene that produced them: the geometry, materials, lighting, camera, and in some formulations the motion and physical properties of objects, stated in terms a graphics renderer could use to reproduce the observation. Rendering, the forward direction, maps a scene description to an image. Inverse graphics runs the other way. Matthew Loper and Michael Black defined it in 2014 as an attempt "to take sensor data and infer 3D geometry, illumination, materials, and motions such that a graphics renderer could realistically reproduce the observed scene", and noted that renderers "are designed to solve the forward process of image synthesis" and "are not usually built to be inverted".^[1] The idea is older than deep learning. It descends from the view of computer vision as Bayesian inference and "analysis by synthesis", and it now spans probabilistic programs, neural networks trained to invert renderers, differentiable renderers, neural scene representations such as NeRF and Gaussian splatting, and, since the arrival of capable large language models, agents that write scene code for tools such as Blender, render it, and revise it until the render matches a target.^[2][3]

Formulation

In the simplest statement, a forward rendering function f takes scene parameters Θ and returns an image. Loper and Black write the basic objective as finding the parameters that minimize the difference between the rendered image and the observed image I, E(Θ) = ||f(Θ) - I||², and note that practical systems add robust penalties and priors while keeping the same core idea: "minimize the difference between the synthesized and observed data".^[1] They describe the approach as trying to "reverse-engineer the physical process that produced an image of the world".^[1]

The inversion is ill-posed. Many different scenes can produce the same image, and some quantities are lost in image formation altogether. Alan Yuille and Daniel Kersten, in a 2006 review in Trends in Cognitive Sciences, framed vision as "an inverse inference problem, in the spirit of Helmholtz, where the goal is to estimate the factors that have generated the image", resolved by placing probability distributions on structured scene representations (Bayesian inference).^[4] The gradSim authors make the same point about physical parameters: estimating mass, friction, and elasticity from video is "fundamentally ill-posed due to the loss of information during image formation".^[5]

Analysis by synthesis

Yuille and Kersten argue that treating vision as inverse inference suggests an algorithm that combines a top-down generative model with bottom-up processing. The generative component lets a system "internally simulate, or synthesize" images, which is why the strategy is called analysis by synthesis.^[4] In their version, low-level cues and grouping rules make fast bottom-up proposals about objects and scene structure, and those hypotheses are then accepted or rejected "by direct comparison with the image (or a filtered version of it) in a top-down process".^[4] They also stress that analysis by synthesis "is not necessarily required for Bayesian inference", and they make a separate case for why a visual system would use it.^[4]

History

Loper and Black trace computer vision as analysis by synthesis to Ulf Grenander's pattern theory, citing his Lectures in Pattern Theory (1976 to 1981) and David Mumford's 1994 work on neuronal architectures for pattern-theoretic problems, and they cite Bruce Baumgart's 1974 Stanford AI Lab memo "Geometric modeling for computer vision" and a 1989 SIGGRAPH panel paper by Demetri Terzopoulos for the use of the term inverse graphics.^[1] Mumford has written that one of the ideas that convinced him Grenander's pattern theory was "the right way to understand almost all cognitive skills and especially vision" was "the emphasis on pattern synthesis as well as pattern analysis".^[6]

Neural-network research in the early 2010s moved in a related direction. In "Transforming Auto-encoders" (ICANN 2011), Geoffrey Hinton, Alex Krizhevsky and Sida Wang argued that networks should use local "capsules" that output a vector of "instantiation parameters", such as pose, lighting and deformation, and that if a capsule's pose output is "linearly related to the 'natural' representations of pose used in computer graphics", checking spatial relationships between parts becomes simple.^[7] Their preliminary 3D experiments used computer graphics to generate stereo images of cars from many viewpoints.^[7]

Two 2015 systems built the inverse-graphics approach into working software. Picture, a probabilistic programming language for scene perception presented at CVPR 2015 by Tejas Kulkarni, Pushmeet Kohli, Joshua Tenenbaum and Vikash Mansinghka, let researchers write generative models of 2D and 3D scenes and solve them with general-purpose inference that mixed Monte Carlo methods with bottom-up, data-driven proposals; its authors used it for 3D face analysis, 3D human pose estimation, and 3D object reconstruction.^[8] The Deep Convolutional Inverse Graphics Network (DC-IGN), by Kulkarni, William Whitney, Kohli and Tenenbaum at NeurIPS 2015, trained a convolutional encoder-decoder with Stochastic Gradient Variational Bayes (the training method of the variational autoencoder) and encouraged neurons in a "graphics code" layer to represent specific transformations such as pose or light, so that from a single image the model could generate the same object under new poses and lighting.^[9]

Main approaches

Learned inverse networks

Sampling-based analysis by synthesis is flexible but slow, and much work has tried to amortize it with a trained network. In Neural Scene De-rendering (CVPR 2017), Jiajun Wu, Tenenbaum and Kohli used "a deterministic rendering function as the decoder", mapping a structured scene description they called scene XML to an image, so that the encoder "is forced to perform the inverse of the rendering operation (a.k.a. de-rendering)".^[11] A 2020 paper in Science Advances by Ilker Yildirim, Mario Belledonne, Winrich Freiwald and Tenenbaum noted that mechanistic analysis-by-synthesis implementations "have typically been too slow for online perception" and proposed an "efficient inverse graphics" model: a deep network that learns to invert a 3D face graphics program "in a single fast feedforward pass".^[12] The authors report that the model accounts for human behavior, including the hollow-face illusion, and maps onto a face-processing circuit in the primate brain.^[12]

Differentiable rendering

A renderer can be inverted by gradient descent only if its output can be differentiated with respect to its inputs. OpenDR, released by Loper and Black in 2014, was built on its own automatic differentiation package and OpenGL; it let users express a forward graphics model and "automatically obtain derivatives with respect to the model parameters", and the authors demonstrated it by estimating human body shape from Kinect depth and RGB data.^[1]

Rasterization is the main obstacle. Hiroharu Kato, Yoshitaka Ushiku and Tatsuya Harada's Neural 3D Mesh Renderer (CVPR 2018) observed that rasterization is a discrete operation "which prevents back-propagation" and proposed an approximate gradient for it, using the renderer for single-image mesh reconstruction with silhouette supervision.^[13] Soft Rasterizer (ICCV 2019) by Shichen Liu, Tianye Li, Weikai Chen and Hao Li instead reformulated rendering as an aggregation of probabilistic contributions from all mesh triangles, which its authors describe as "truly differentiable" and able to pass gradients to occluded and distant vertices.^[15]

Physically based renderers raised the same problem for visibility. Tzu-Mao Li, Miika Aittala, Frédo Durand and Jaakko Lehtinen's "Differentiable Monte Carlo Ray Tracing through Edge Sampling" (SIGGRAPH Asia 2018) addressed the fact that "the rendering integral includes visibility terms that are not differentiable" by sampling the Dirac delta functions those terms introduce, producing gradients with respect to camera pose, geometry, materials and lighting; the authors interfaced the ray tracer with PyTorch and demonstrated inverse rendering applications.^[14] Mitsuba 2 (SIGGRAPH Asia 2019), by Merlin Nimier-David, Delio Vicini, Tizian Zeltner and Wenzel Jakob, made a physically based renderer "retargetable" so that the same code could be compiled for forward rendering or with forward- and reverse-mode automatic differentiation; one of its demonstrations reconstructed a smoke volume from reference images.^[16] Its successor, Mitsuba 3, is described by EPFL's Realistic Graphics Lab as "a retargetable forward and inverse renderer" built on the Dr.Jit just-in-time compiler.^[17] Recovering scene parameters through a physically based renderer in this way is often called inverse rendering, the term Li and co-authors use for their applications and the sense in which Mitsuba is an "inverse renderer".^[14][17]

Neural scene representations

Neural radiance fields recast the reconstruction of real photographed scenes as optimization of a learned scene representation. NeRF (Mildenhall et al., 2020) represents a scene as a fully connected network that maps a 3D position and viewing direction to density and emitted radiance, and renders images with classical volume rendering; because "volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses".^[18] The paper, whose first author is Ben Mildenhall, applied it to novel view synthesis. 3D Gaussian splatting (Kerbl et al., 2023) replaced the network with a set of optimized anisotropic 3D Gaussians initialized from the sparse points produced during camera calibration, together with a fast visibility-aware rasterizer, reaching real-time (30 frames per second or more) novel-view synthesis at 1080p.^[19] Both methods recover an appearance model that reproduces the input photographs rather than a description in terms of separate objects, materials and light sources.

Program synthesis

Another line treats the scene description as a program. Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama and Tenenbaum (NeurIPS 2018) trained a network to propose drawing primitives from hand drawings and then used program synthesis to recover a graphics program, written in a subset of LaTeX, with loops, variable bindings and simple conditionals; with the program in hand they could correct network errors and extrapolate drawings.^[20] CSGNet (CVPR 2018) by Gopal Sharma and co-authors takes a 2D or 3D shape and outputs a constructive solid geometry program of Boolean operations on primitives, and can be trained without ground-truth programs using policy gradients.^[21] SPIRAL (2018), from Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami and Oriol Vinyals, is an agent that generates a program executed by a graphics engine and is trained with distributed reinforcement learning, using an adversarial discriminator's output as its reward; its authors called it, to their knowledge, the first end-to-end unsupervised adversarial inverse graphics agent on real-world datasets such as MNIST, Omniglot and CelebA.^[22]

CAD reconstruction is a close relative. CAD-Recode (2024) represents sketch-and-extrude CAD sequences as Python code and uses a small language model as a decoder that translates a point cloud into code which, when executed, rebuilds the CAD model.^[25]

Inverse physics

Recovering how a scene moves, and why, extends inverse graphics from appearance to dynamics. Galileo (NeurIPS 2015), by Jiajun Wu, Ilker Yildirim, Joseph Lim, Bill Freeman and Tenenbaum, put a 3D physics engine at the core of a generative model over objects' mass, position, 3D shape and friction, inferred those properties with short runs of MCMC that drive simulations to fit visual observations, and reported accuracy comparable to human subjects on simple but physically rich videos.^[10] "Learning to See Physics via Visual De-animation" (NeurIPS 2017) recovered a physical world state with a perception module and then used physics and graphics engines for reasoning and prediction; its authors note that "even more so than forward simulation, inverting a physics or graphics engine is a computationally hard problem" and used a convolutional inversion network to make it tractable.^[26]

Differentiable simulators allow gradient-based inversion of dynamics in the same way differentiable renderers do for appearance. DiffTaichi (ICLR 2020) is a differentiable programming language for building physical simulators that generates gradients of simulation steps by source-code transformation.^[27] gradSim (ICLR 2021) combined differentiable multiphysics simulation with differentiable rendering so that gradients flow "from pixels in a video sequence through to the underlying physical attributes that generated them", estimating properties such as mass, friction and elasticity without 3D supervision.^[5]

Language-model agents and scene code

Large language models brought a different inversion method: write the scene as code in an existing 3D tool, render it, look at the result, and edit the code. Peter Kulits and co-authors, including Michael Black, defined inverse graphics as "the task of inverting an image into physical variables that, when rendered, enable reproduction of the observed scene" and proposed IG-LLM, which decodes a visual embedding into a structured, compositional 3D scene representation through next-token prediction, "without the application of image-space supervision".^[28] 3D-GPT (2023) used several LLM agents to turn text instructions into parameters for procedural 3D generation and integrated with Blender.^[29]

Several systems close the render-and-compare loop explicitly:

In SceneCraft, the vision language model that inspects each render plays the role that top-down verification plays in analysis by synthesis: checking a hypothesized scene against an image.^[2][4] BlenderGym's authors also studied inference-time scaling and found that the verifier guiding generation can itself be improved by spending more inference compute, and that compute is best split deliberately between generation and verification.^[24]

The same loop appears in CAD. BenchCAD (May 2026), from Haozhe Zhang and co-authors, contains 17,900 execution-verified CadQuery programs across 106 industrial part families; its Vision2Code task asks a model to write executable CadQuery code from four canonical orthographic views and scores the result by IoU against the ground-truth shape.^[30] Across more than ten frontier models, the authors found that systems "often recover coarse outer geometry but fail to produce faithful parametric CAD programs", for example replacing sweeps, lofts and twist-extrudes with simpler sketch-and-extrude patterns.^[30]

GPT-6 Astra

OpenAI's September 3, 2026 launch post for GPT-6 Astra included 3D work among its demonstrations, although it did not use the phrase inverse graphics. The post says Astra "brings stronger visual judgment to the websites, games, applications, and renderings it builds", and its demonstrations show Astra modeling a house in Blender and turning it into "a walkable scene in Unreal Engine 5", and turning "a spaceship concept into a 3D model".^[31] Among its professional benchmarks, OpenAI describes BenchCAD as testing "whether models can reconstruct 3D objects from multi-view renders by generating CAD code", and reports that with tools Astra reached a 95.9% geometric-overlap score, against 83.3% for GPT-5.6 Sol and 84.3% reported for Claude Fable 5.1; OpenAI notes that the Claude scores reflect three modifications to the evaluation described in Anthropic's own system card.^[31] The New Stack reported that the Astra figure was measured on BenchCAD's 1,000-file Vision2Code subset with Python tools, a sample size OpenAI's post does not state.^[32] Anthropic's Fable 5.1 system card says its Claude scores come from a random 1,000-file subset, while the GPT-5.6 Sol scores it compared against were OpenAI's, evaluated on the full 17,900 files.^[33] These are company-reported results.

Limitations and open problems

• Ambiguity. Images underdetermine scenes, so any inversion depends on priors, whether hand-built as in Bayesian models or learned.^[4] Some physical parameters are hard to recover from images alone, which is why gradSim describes physical system identification from video as ill-posed.^[5]
• Speed. Sampling-based analysis by synthesis has typically been too slow for online perception, the problem Yildirim and colleagues set out to address with a single feedforward inverse network.^[12]
• Discontinuities. Rasterization and visibility make rendering non-differentiable, which each differentiable renderer handles with an approximation, a smoothed formulation, or specialized sampling.^[13][14][15]
• Structure versus appearance. Radiance fields and Gaussian splats reproduce photographs well but do not by themselves yield object-level, editable scene descriptions.^[18][19]
• Precision of code-based reconstruction. Benchmarks built around code, such as BlenderGym and BenchCAD, report that current models handle coarse structure better than exact edits or faithful parametric programs.^[24][30]

References

1. ^1 ^2 ^3 ^4 ^5 ^6OpenDR: An Approximate Differentiable Renderer - Matthew M. Loper and Michael J. Black, ECCV 2014, pp. 154-169 (author PDF: files.is.tue.mpg.de/...OpenDR.pdf).
2. ^1 ^2 ^3 ^4SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code - Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A. Ross, Cordelia Schmid, Alireza Fathi, arXiv:2403.01248, 2024.
3. ^1 ^2 ^3LL3M: Large Language 3D Modelers - Sining Lu, Guan Chen, Nam Anh Dinh, Itai Lang, Ari Holtzman, Rana Hanocka, arXiv:2508.08228, 2025.
4. ^1 ^2 ^3 ^4 ^5 ^6Vision as Bayesian inference: analysis by synthesis? - Alan Yuille and Daniel Kersten, Trends in Cognitive Sciences 10(7):301-308, July 2006.
5. ^1 ^2 ^3gradSim: Differentiable simulation for system identification and visuomotor control - Krishna Murthy Jatavallabhula et al., arXiv:2104.02646 (ICLR 2021).
6. ^Work in Pattern Theory - David Mumford, personal archive, Brown University (accessed September 23, 2026).
7. ^1 ^2Transforming Auto-Encoders - Geoffrey E. Hinton, Alex Krizhevsky, Sida D. Wang, International Conference on Artificial Neural Networks (ICANN) 2011, pp. 44-51.
8. ^1 ^2Picture: A Probabilistic Programming Language for Scene Perception - Tejas D. Kulkarni, Pushmeet Kohli, Joshua B. Tenenbaum, Vikash Mansinghka, CVPR 2015, pp. 4390-4399.
9. ^1 ^2Deep Convolutional Inverse Graphics Network - Tejas D. Kulkarni, William F. Whitney, Pushmeet Kohli, Joshua B. Tenenbaum, NeurIPS 2015 (arXiv:1503.03167).
10. ^1 ^2Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning - Jiajun Wu, Ilker Yildirim, Joseph J. Lim, Bill Freeman, Josh Tenenbaum, NeurIPS 2015.
11. ^1 ^2Neural Scene De-Rendering - Jiajun Wu, Joshua B. Tenenbaum, Pushmeet Kohli, CVPR 2017, pp. 699-707.
12. ^1 ^2 ^3 ^4Efficient inverse graphics in biological face processing - Ilker Yildirim, Mario Belledonne, Winrich Freiwald, Josh Tenenbaum, Science Advances 6(10):eaax5979, 2020.
13. ^1 ^2 ^3Neural 3D Mesh Renderer - Hiroharu Kato, Yoshitaka Ushiku, Tatsuya Harada, CVPR 2018, pp. 3907-3916 (arXiv:1711.07566).
14. ^1 ^2 ^3 ^4Differentiable Monte Carlo Ray Tracing through Edge Sampling - Tzu-Mao Li, Miika Aittala, Frédo Durand, Jaakko Lehtinen, ACM Transactions on Graphics 37(6) (SIGGRAPH Asia 2018).
15. ^1 ^2 ^3Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning - Shichen Liu, Tianye Li, Weikai Chen, Hao Li, ICCV 2019, pp. 7708-7717 (arXiv:1904.01786).
16. ^1 ^2Mitsuba 2: A Retargetable Forward and Inverse Renderer - Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Wenzel Jakob, ACM Transactions on Graphics 38(6) (SIGGRAPH Asia 2019), doi:10.1145/3355089.3356498.
17. ^1 ^2 ^3Mitsuba 3: A Retargetable Forward and Inverse Renderer - Realistic Graphics Lab, EPFL, project site (accessed September 23, 2026).
18. ^1 ^2 ^3NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis - Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, arXiv:2003.08934, 2020 (ECCV 2020).
19. ^1 ^2 ^33D Gaussian Splatting for Real-Time Radiance Field Rendering - Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis, ACM Transactions on Graphics 42(4), July 2023 (arXiv:2308.04079).
20. ^1 ^2Learning to Infer Graphics Programs from Hand-Drawn Images - Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, Joshua B. Tenenbaum, NeurIPS 2018 (arXiv:1707.09627).
21. ^1 ^2CSGNet: Neural Shape Parser for Constructive Solid Geometry - Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, Subhransu Maji, arXiv:1712.08290 (CVPR 2018).
22. ^1 ^2Synthesizing Programs for Images using Reinforced Adversarial Learning - Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami, Oriol Vinyals, arXiv:1804.01118, 2018.
23. ^1 ^2BlenderAlchemy: Editing 3D Graphics with Vision-Language Models - Ian Huang, Guandao Yang, Leonidas Guibas, arXiv:2404.17672, 2024.
24. ^1 ^2 ^3 ^4BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing - Yunqi Gu, Ian Huang, Jihyeon Je, Guandao Yang, Leonidas Guibas, arXiv:2504.01786 (CVPR 2025).
25. ^CAD-Recode: Reverse Engineering CAD Code from Point Clouds - Danila Rukhovich, Elona Dupont, Dimitrios Mallis, Kseniya Cherenkova, Anis Kacem, Djamila Aouada, arXiv:2412.14042, 2024.
26. ^Learning to See Physics via Visual De-animation - Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, Josh Tenenbaum, NeurIPS 2017.
27. ^DiffTaichi: Differentiable Programming for Physical Simulation - Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, Frédo Durand, arXiv:1910.00935 (ICLR 2020).
28. ^Re-Thinking Inverse Graphics With Large Language Models - Peter Kulits, Haiwen Feng, Weiyang Liu, Victoria Abrevaya, Michael J. Black, arXiv:2404.15228, 2024 (Transactions on Machine Learning Research).
29. ^3D-GPT: Procedural 3D Modeling with Large Language Models - Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, Stephen Gould, arXiv:2310.12945, 2023.
30. ^1 ^2 ^3BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD - Haozhe Zhang, Kaichen Liu, Miaomiao Chen, Lei Li, Shaojie Yang, Cheng Peng, Hanjie Chen, arXiv:2605.10865, May 2026.
31. ^1 ^2GPT-6 Astra: A new generation of intelligence - OpenAI, September 3, 2026 (text checked against the Internet Archive snapshot of September 20, 2026).
32. ^OpenAI launches GPT-6 Astra and says welcome to the "AGI era" - The New Stack (Frederic Lardinois), September 3, 2026.
33. ^System Card: Claude Fable 5.1 & Claude Mythos 5.1 - Anthropic, September 1, 2026, section 8.14.2 BenchCAD.