Radiometric Consensus: Disentangling Materials and Illumination Through Multi-Object Constraints

Inverse rendering, the recovery of physical scene attributes from images, represents one of computer vision's most persistent challenges. When constrained to a single observation, the problem becomes mathematically underdetermined due to the information loss inherent in the rendering process. Surface reflectance acts as a low pass filter on incident illumination, destroying high frequency lighting details that cannot be recovered through optimization alone. Traditional approaches have addressed this ambiguity through deterministic regression or iterative optimization, yielding single point estimates that ignore the fundamentally multimodal nature of the solution space. The paper "Under One Sun: Multi-Object Generative Perception of Materials and Illumination" introduces MultiGP, a method that reframes this challenge through the lens of generative perception, leveraging multi object consensus to sample from the distribution of plausible radiometric explanations including texture, reflectance, and shared illumination.

The Architecture of Physical Consistency

The technical innovation of MultiGP rests on four interconnected architectural contributions that collectively enforce physical consistency across object instances. First, the authors propose a cascaded end to end architecture that operates simultaneously in image space and angular space. This dual domain approach disentangles texture, which varies across surface positions, from reflectance and illumination, which interact in the angular domain of the reflectance map. By handling texture extraction in the image domain while resolving BRDFs and lighting in the angular domain, the system preserves the distinct frequency characteristics of each radiometric constituent. This separation is crucial because texture and illumination occupy different frequency bands; conflating their estimation leads to the classic ambiguity where texture patterns are mistaken for lighting effects.

Second, and most critically, the method introduces Coordinated Guidance to manage the temporal dynamics of the diffusion process. In standard diffusion models, each object would be processed independently, likely producing inconsistent illumination estimates that violate physical reality. Coordinated Guidance implements a scheduling mechanism that synchronizes the denoising trajectories across multiple objects, forcing the sampling process to converge toward a single shared environment map. This directly encodes the physical constraint that all objects in a scene exist under one sun. The technical implementation involves careful orchestration of the noise levels and sampling steps across object specific diffusion paths, ensuring that illumination estimates remain coupled throughout the generative process rather than diverging into incompatible explanations.

Third, Axial Attention facilitates information exchange between objects within the reflectance map domain. Since different materials filter incident light differently according to their BRDF properties, one object may preserve angular frequency information that another has attenuated. A diffuse object captures little information about the directional distribution of light, while a specular object captures detailed reflections of the environment. Axial Attention allows these complementary observations to inform one another, effectively enabling objects with glossy surfaces to assist in reconstructing illumination details for matte objects, and vice versa. This cross object communication happens in the angular space of the reflectance maps, allowing the network to pool partial information across material types to reconstruct a more complete lighting estimate than any single object could provide alone.

Fourth, the Texture Extraction ControlNet ensures that high frequency surface details are preserved without leaking into the illumination estimate. This addresses a common failure mode in inverse rendering, where albedo variations are mistakenly attributed to shading. By conditioning the texture extraction on the estimated illumination while maintaining architectural separation through the ControlNet structure, the system achieves genuine decoupling of surface appearance from lighting. The ControlNet architecture allows the model to preserve fine geometric details and printed patterns while ensuring these remain anchored to the surface reflectance rather than being absorbed into the environment map.

Generative Perception Beyond Point Estimates

MultiGP distinguishes itself from prior work through its commitment to stochastic sampling rather than deterministic prediction. Previous methods such as DRM could generate samples of reflectance and illumination, but were limited to textureless objects, severely restricting their applicability to real world scenes. Deterministic regression approaches, including many deep learning based inverse rendering systems, produce single "best fit