Inverse Graphics Without Differentiable Rendering: What Staged Reconstruction Tells Us About VLM Priors

Inverse graphics has been one of the oldest aspirations in computer vision, going back to Roberts' "Blocks World" in 1963. The goal is deceptively simple to state and notoriously hard to achieve: take a single image and recover the editable 3D scene that produced it, including geometry, materials, lighting, and camera. If you can recover that scene, then relighting, novel-view synthesis, and editing become trivial downstream operations rather than separate research problems.

A recent paper, "Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Models" by Guangzhao He, Rundong Luo, Wei-Chiu Ma, and Hadar Averbuch-Elor from Cornell, makes a claim that runs against the grain of the last several years of work in this area. They show that you can reconstruct an editable Blender program directly from a single image using nothing more than a pretrained vision-language model. No differentiable rendering. No multi-view supervision. No specialized 2D or 3D foundation models bolted on. The trick, and it is an important one, is in how you ask.

The Monolithic Failure Mode

The natural way to use a VLM for this task is to show it an image and ask it to output Blender code that reproduces the scene. This is the monolithic formulation, and it largely fails. The reason is intuitive once you think about the structure of the problem. A scene program couples many factors that interact non-linearly in the rendered output. The perceived color of a surface depends jointly on its material albedo, the lighting, and the geometry orienting it toward or away from light sources. Asking a model to nail all of these at once means it must solve a high-dimensional, heavily entangled inverse problem in a single forward pass, with no opportunity to inspect intermediate results.

This is precisely where classical inverse rendering methods historically deployed analysis-by-synthesis: render a hypothesis, compare it to the target, and iterate. Neural approaches replaced the explicit optimization loop with gradients through a differentiable renderer. The Cornell work observes that a VLM, used naively, gets neither the iteration nor the gradient. It is essentially asked to guess the entire factored scene in one shot, which is why it struggles.

Staging as the Core Contribution

The central insight of SEIG (Staged Executable Inverse Graphics) is to decompose reconstruction into sequential, semantically meaningful stages that mirror how professional 3D artists actually work. Artists do not configure geometry, materials, composition, and lighting simultaneously. They build a coarse scaffold, then refine one factor at a time, continuously rendering intermediate states and adjusting until the result matches their target.

SEIG operationalizes this as an agentic generator-verifier loop:

• Initialization. From the input image, the system creates a coarse scene scaffold of simple geometric primitives with approximate object layouts.
• Sequential refinement. Successive stages recover geometry, materials, object composition, and lighting, each expressed directly as executable Blender code.
• Verification. Each stage is paired with a verifier module that renders the current scene and evaluates it before guiding the next refinement.

The design has two properties worth emphasizing. First, the representation stays a structured, physically grounded Blender program throughout, never collapsing into an opaque neural latent. That means the output is editable at every intermediate step, not just at the end. Second, the verifier reintroduces the analysis-by-synthesis loop that monolithic generation throws away, but it does so in pixel space using the VLM's own visual judgment rather than through gradients.

The authors evaluate across synthetic and in-the-wild scenes using metrics spanning three levels of fidelity: pixel-level, perceptual, and semantic. The headline empirical result is that staged reconstruction substantially improves fidelity over monolithic baselines, and crucially this holds even against baselines equipped with specialist 2D and 3D foundation models. In other words, decomposition of the task contributes more than the richness of the external toolkit. That is a strong claim, and it is the most interesting finding in the paper.

Why This Result Matters Beyond Blender

The deeper implication is about where the bottleneck actually lives. For years the implicit assumption has been that single-image inverse graphics fails because models lack sufficient 3D knowledge, which motivated training large 3D foundation models and constructing differentiable pipelines to inject geometric priors. The SEIG results suggest the latent 3D priors were already present inside pretrained VLMs. What was missing was a problem decomposition that let those priors be expressed without forcing the model to solve a fully entangled inverse problem at once.

This connects to a broader pattern we keep observing across reasoning-heavy tasks: large models often possess the requisite knowledge but fail when forced to produce a complex structured answer in a single step. Chain-of-thought prompting, agentic tool use, and verifier-guided refinement are all variations on the same theme of trading a hard one-shot problem for a sequence of easier conditioned subproblems. SEIG is essentially chain-of-thought for graphics programs, where each "thought" is an executable, renderable, inspectable scene state.

There is also an elegant alignment between the method and the artifact it produces. Because the output is a Blender program rather than a neural field like a NeRF or a 3D Gaussian splat, downstream applications come almost for free. Relighting is editing the light nodes. Novel views are moving the camera. Editing is changing object parameters. Physics simulation runs on the explicit geometry. Neural representations, by contrast, entangle geometry, materials, and illumination in weights that resist direct manipulation, and even disentangled variants rarely yield a clean editable program. SEIG sidesteps that entire class of problems by committing to symbolic, executable output from the start.

Limitations and Open Questions

I want to be careful not to overstate what this demonstrates. A few caveats stand out.

• Complexity ceiling. Reconstruction from primitives and procedural Blender code is well suited to scenes that decompose into recognizable objects with relatively clean materials. Highly detailed organic geometry, complex translucency, subsurface scattering, and intricate texture detail are likely to stress the approach. The paper's strongest results probably live in the regime of compositional, object-centric scenes.
• Verifier reliability. The whole loop depends on the VLM's ability to compare a render against the target and identify what to fix. VLM visual judgment is known to be unreliable on fine-grained perceptual differences, especially subtle lighting and material mismatches. The quality of the verifier is likely the true determinant of final fidelity, and it would be valuable to see ablations isolating verifier accuracy.
• Compute and latency. A staged generator-verifier loop with repeated rendering and VLM calls is far more expensive than a single forward pass. For interactive or large-scale use, the cost structure matters.
• Metric interpretation. Pixel, perceptual, and semantic metrics each tell a partial story. A scene can be semantically faithful while being pixel-wise far off, and vice versa. The claim that decomposition beats toolkit richness is most convincing if it holds across all three levels consistently.

Closing Thoughts

What I find most compelling about this work is the reframing of the bottleneck. The community spent considerable effort building heavy machinery to inject 3D knowledge into inverse graphics, when at least part of that knowledge was already latent in general-purpose VLMs, waiting for the right interface. The interface turned out to be the artist's own iterative workflow, encoded as staged executable refinement.

This raises several questions worth pursuing. How far up the complexity ladder can staging carry us before the VLM's priors genuinely run out? Could the verifier be improved by pairing the VLM with lightweight perceptual critics for the factors where it is weakest, particularly lighting? And more broadly, how many other underconstrained vision tasks are secretly bottlenecked not by missing knowledge but by the demand to produce everything in one shot? If the SEIG result generalizes, the lesson is that decomposition matched to expert workflow may be a more reliable lever than ever-larger specialist models.