Disentangling Light from Matter: Appearance Decomposition in Multi-Traversal Neural Reconstruction

The construction of photorealistic digital twins for autonomous driving simulation presents a fundamental tension in computer vision. We desire geometric consistency across multiple captures of the same location, yet we must accommodate dramatic appearance variations caused by changing illumination, weather conditions, and seasonal solar angles. The recent preprint Appearance Decomposition Gaussian Splatting for Multi-Traversal Reconstruction by Xiao et al. addresses this tension through a principled decomposition of appearance into traversal invariant material properties and traversal dependent illumination. Their framework, ADM-GS, moves beyond entangled latent representations to offer a more structured approach to multi-traversal scene reconstruction.

The Entanglement Problem in Latent-Based Reconstruction

Existing approaches to multi-traversal reconstruction, including methods like 4DGF and MTGS, typically rely on scene graphs that separate static backgrounds from dynamic objects. These frameworks model appearance variations across traversals using environment latent codes or per-traversal color residuals. While effective for view synthesis within individual traversals, this latent-based paradigm suffers from a critical limitation: high frequency lighting effects, including cast shadows and specular highlights on vehicle surfaces, become absorbed into the static appearance representation.

This entanglement creates significant artifacts when the model attempts to synthesize views under large cross-traversal lighting variations. When illumination changes are encoded implicitly within neural network weights or latent vectors, the system cannot distinguish between stable material properties, such as the albedo of road surfaces or building facades, and transient lighting conditions. The result is a brittle representation that struggles to generalize across the diurnal and seasonal cycles characteristic of long-term autonomous driving datasets.

The technical challenge is exacerbated by the inherent properties of 3D Gaussian Splatting (3DGS), the underlying representation employed by current state-of-the-art methods. Unlike neural radiance fields (NeRF) which implicitly represent surfaces through density fields, 3DGS utilizes anisotropic ellipsoids for volumetric filling. This design lacks explicit surface normals, making normal-guided illumination estimation unstable without additional geometric regularization.

Technical Architecture and Frequency Separation

ADM-GS addresses these limitations through an explicit appearance decomposition framework that separates traversal-invariant material from traversal-dependent illumination. For the static background, the final color is computed as the product of a material field M and an illumination field L_m, where M captures intrinsic albedo and surface properties while L_m models lighting variations specific to each traversal.

To implement this decomposition within 3DGS, the authors introduce several key technical innovations. First, they apply geometric regularization to stabilize surface-aware illumination estimation. By enforcing consistency in the estimated normals across the Gaussian primitives, they create a stable substrate for lighting calculations that the base 3DGS representation does not naturally provide.

Second, and more notably, they propose a neural light field utilizing a frequency-separated hybrid encoding strategy. This design explicitly separates low-frequency diffuse illumination from high-frequency specular reflections. By incorporating surface normals and explicit reflection vectors, the network can model view-dependent specular effects on car bodies and glass surfaces without conflating these transient highlights with the underlying material appearance. This frequency-aware approach proves crucial in complex outdoor scenes where glossy vehicle surfaces and matte road textures coexist.

The framework also adopts transient geometry embeddings from prior work to handle regions where geometry itself changes across traversals, such as construction zones or parked vehicles. This allows the material-illumination decomposition to focus on truly static background elements while relegating geometric inconsistencies to a separate handling mechanism.

Evaluation and Broader Implications

Quantitative evaluations on the Argoverse 2 and Waymo Open datasets demonstrate the effectiveness of this approach. ADM-GS achieves a +0.98 dB PSNR improvement over existing latent-based baselines in multi-traversal experiments. Beyond raw metrics, the method produces qualitatively superior results with more consistent material appearance across varying environmental conditions. The decomposition successfully preserves the stable visual properties of scene elements while accurately capturing traversal-specific illumination variations.

These results suggest a broader shift in neural scene reconstruction toward explicit physical priors. While early neural rendering methods prioritized representational flexibility, often learning entangled mappings from position and viewing direction to color, recent work increasingly recognizes the value of structured decomposition. By incorporating concepts from inverse rendering, such as material-illumination separation, ADM-GS represents a convergence between traditional computer graphics pipelines and modern differentiable rendering techniques.

Limitations and Future Directions

Despite its advantages, the ADM-GS framework raises several questions that point toward future research. The current formulation assumes that material properties remain strictly invariant across traversals. While appropriate for urban infrastructure such as roads and buildings, this assumption breaks down for natural environments where vegetation changes seasonally or for man-made surfaces that weather and degrade over time. A more sophisticated model might incorporate temporal material variation as a third decomposition component, distinct from both static material and transient illumination.

Furthermore, the computational overhead of maintaining separate neural networks for material and illumination fields, combined with geometric regularization, may limit scalability to city-scale reconstructions. The frequency-separated encoding, while effective, adds architectural complexity that could complicate optimization in highly heterogeneous environments.

From a methodological perspective, the reliance on estimated surface normals within 3DGS remains a source of potential instability. While geometric regularization helps, the fundamental mismatch between volumetric Gaussian primitives and explicit surface representations suggests that future work might benefit from hybrid representations that natively support surface-aware rendering.

Conclusion

The ADM-GS framework offers a compelling solution to the appearance inconsistency problem in multi-traversal reconstruction. By explicitly decomposing scene appearance into traversal-invariant material and traversal-dependent illumination, Xiao et al. demonstrate that structured physical priors can outperform entangled latent representations. The frequency-separated neural light field and geometry-guided illumination modeling provide concrete technical mechanisms for handling the complex lighting environments characteristic of autonomous driving scenarios.

As the field moves toward large-scale digital twin construction, the principles underlying ADM-GS, physical decomposition, frequency-aware encoding, and geometric regularization, will likely become standard components of neural reconstruction pipelines. Open questions remain regarding the handling of temporal material changes and the scalability of explicit decomposition frameworks, but the trajectory toward disentangled, interpretable scene representations appears clear. Future work might explore semantic material pri