Surface Light Field Tokenization: Learning 3D Appearance Through Strategic Subsampling

Introduction

The synthesis of realistic 3D objects requires more than accurate geometry. Real materials interact with light in complex ways, producing view-dependent effects such as specular highlights, Fresnel reflections, and subtle color shifts that change with viewing angle. Most existing machine learning approaches to 3D generation have treated appearance as a secondary concern, either focusing exclusively on geometry reconstruction or assuming view-independent diffuse albedo. This limitation produces models that look flat or plastic, lacking the visual richness of real-world surfaces.

In "LiTo: Surface Light Field Tokenization," Chang et al. (ICLR 2026) address this gap by introducing a latent representation that jointly encodes both 3D geometry and view-dependent appearance. Their key insight reframes the problem: rather than treating multiview RGB-D images as independent observations, they represent these as random samples of an underlying surface light field. By training an encoder to compress sparse subsamples of this high-dimensional signal into a compact latent space, the model learns to interpolate missing view-dependent information, effectively capturing material properties without explicitly storing every possible camera angle and lighting condition.

Encoding the Surface Light Field

The technical architecture of LiTo centers on the surface light field, a 4D function describing radiance as a function of position on a surface and viewing direction. Previous approaches typically encoded geometry alone using signed distance functions or occupancy grids, or they captured appearance as static diffuse color independent of viewpoint. LiTo instead encodes tuples of surface position, viewing direction, and observed RGB value, forcing the model to learn the relationship between geometry and material response.

The training strategy employs random subsampling of this light field data. Given RGB-D multiview images, the model samples random subsets of surface points and viewing directions rather than processing complete dense representations. This approach serves two purposes. First, it reduces memory requirements, allowing the architecture to scale to one million input tokens through localized attention patterns. Second, and more importantly, it prevents the model from memorizing specific viewpoints. By reconstructing the full light field from incomplete samples, the encoder must learn the underlying physical structure of surface reflectance, including specular lobes and Fresnel effects.

The decoder translates these latent representations into renderable 3D assets using Gaussian splats augmented with higher-order spherical harmonics. This choice matters because spherical harmonics provide the angular frequency components necessary to represent complex view-dependent lighting. The combination of compact latents with expressive decoding enables LiTo to reproduce reflective surfaces that prior latent representations, such as those using view-independent texture maps, cannot capture.

Latent Flow Matching for Conditional Generation

Beyond reconstruction, LiTo enables generative modeling through a latent flow matching model trained on the learned representation. Flow matching, which has gained traction in image generation, learns continuous normalizing flows by regressing vector fields that transform simple prior distributions into complex data distributions. In this context, the model learns the distribution of 3D latent representations conditioned on single input images.

This conditional generation pipeline addresses a challenging inverse problem: inferring full 3D geometry and material properties from a single 2D observation. The model must not only hallucinate occluded geometry but also infer view-dependent appearance consistent with the lighting and materials visible in the input. By operating in the compressed latent space rather than raw pixel or voxel space, the flow matching model achieves greater efficiency while maintaining the ability to generate diverse samples that respect the input image constraints.

The training framework jointly supervises geometry and appearance, ensuring that generated objects maintain geometric fidelity while displaying appropriate material characteristics. This joint supervision contrasts with sequential approaches that first generate geometry then texture it, which often produce inconsistencies between shape and surface properties.

Your Take: From Storage to Synthesis

The LiTo framework represents a conceptual shift in how we handle high-dimensional continuous signals. Traditional approaches to view-dependent appearance, such as image-based rendering or light field cameras, rely on dense sampling and storage of viewing directions. LiTo inverts this paradigm; it stores only sparse samples and learns the structure necessary for synthesis. This shift from storage to synthesis mirrors broader trends in neural compression and implicit neural representations, where the model becomes a compression algorithm trained to reconstruct signals from minimal information.

The random subsampling strategy deserves particular attention. By preventing the model from seeing complete light fields during training, the authors force interpolation rather than memorization. This heuristic applies beyond computer graphics. In video compression, scientific imaging, and audio processing, we face similar curses of dimensionality where brute force sampling becomes infeasible. LiTo suggests that strategic subsampling combined with learned reconstruction outperforms dense storage, provided the model architecture possesses sufficient inductive bias about the signal structure.

However, the approach carries limitations. The reliance on RGB-D multiview images for training restricts the method to object categories where such data is available, potentially limiting generalization to rare materials or complex translucent objects not well represented in training sets. Additionally, while localized attention improves efficiency, processing one million tokens remains computationally intensive compared to sparse voxel or point-based alternatives. The method also assumes static scenes under fixed lighting during training, leaving open questions about handling dynamic illumination or temporal changes in material properties.

Conclusion

LiTo advances 3D representation learning by unifying geometry and view-dependent appearance in a single latent framework. By encoding surface light fields through strategic subsampling, the model captures complex material behaviors including specular highlights and Fresnel reflections that previous approaches miss. The latent flow matching extension enables practical applications in single-image 3D reconstruction while maintaining visual