The Compositional Tokenization of Light: Why Disentangled Control Matters in Generative Image Relighting

Lighting defines the visual world. It sculpts form, directs attention, and establishes mood. Yet once a photograph is captured, the lighting is fixed, immutable. Traditional post-production offers limited recourse: global brightness adjustments or color grading that treats illumination as a uniform layer. True relighting, the ability to place a virtual light source behind a subject or toggle a specific lamp within the scene, has remained the exclusive domain of 3D rendering pipelines, requiring precise geometry, material estimation, and physics simulation.

TokenLight: Precise Lighting Control in Images using Attribute Tokens, presented by Chaturvedi et al., challenges this dichotomy by demonstrating that diffusion models can learn intricate light transport physics without explicit inverse rendering. The key insight lies not in architectural novelty per se, but in a fundamental reconsideration of how we encode physical phenomena for generative models. By tokenizing continuous lighting attributes into discrete, semantically meaningful units rather than compressing them into monolithic embeddings, the authors achieve a level of precise, compositional control that previous text-driven or image-based methods have struggled to attain.

The Architecture of Disentangled Control

At its core, TokenLight formulates relighting as a conditional image generation task. The model accepts a source image and a set of lighting specifications, then re-renders the scene under the new illumination. Where this work diverges from prior diffusion-based approaches is in the conditioning mechanism. Rather than encoding lighting parameters into a single concatenated vector or relying on vague text descriptions like "add dramatic side lighting," TokenLight employs attribute tokens: distinct, interpretable embeddings that represent intensity, color (RGB values), ambient illumination levels, diffuse coefficients, and crucially, 3D spatial coordinates (x, y, z) of light sources.

This tokenized representation is fed into a diffusion transformer architecture, specifically fine-tuned from a foundation model pretrained on broad text-to-image and text-to-video tasks. The training data consists primarily of synthetically rendered images using Blender's path-tracing engine, where ground-truth lighting annotations are exact and systematically varied. Each training pair consists of the same scene rendered under different combinations of the tokenized attributes, allowing the model to learn the mapping between specific parameter changes and pixel-level appearance shifts.

The synthetic training regimen is supplemented by a smaller corpus of real-world captures where lighting changes correspond to physical light sources being toggled on or off. This hybrid approach addresses the annotation bottleneck that typically plagues relighting research; while real-world datasets with precise lighting ground truth are prohibitively expensive to capture at scale, the synthetic data provides the density of examples necessary for the model to learn continuous, fine-grained control.

Implicit Physics Through Data

Perhaps the most striking aspect of TokenLight is its apparent understanding of physical light interactions despite receiving no explicit supervision regarding scene geometry, surface normals, or material properties. The model demonstrates the ability to place virtual lights inside translucent objects, such as illuminating a pumpkin from within to create a jack-o'-lantern effect, or adding backlighting that correctly respects occlusion boundaries and casts plausible shadows on complex fur geometry.

This capability suggests that large-scale diffusion models, when conditioned on sufficiently factorized physical parameters, can internalize approximate models of light transport, geometric occlusion, and material response. The network learns to infer hidden surfaces and transmissive properties purely from statistical patterns in the training data, effectively performing a form of implicit inverse rendering without the brittle intermediate representations that plague explicit 3D reconstruction pipelines.

The results challenge the prevailing assumption in computer graphics that physically accurate relighting requires explicit scene decomposition into geometry, reflectance, and illumination components. Instead, TokenLight demonstrates an end-to-end control-to-render paradigm where the generative model itself becomes the renderer, bypassing the need for accurate depth maps or normal estimation that often fail on complex materials like glass, hair, or volumetric media.

The Compositional Factorization Principle

The technical success of TokenLight illuminates a broader principle for conditional generation: continuous, multi-attribute phenomena should be decomposed into orthogonal, semantically meaningful control axes before conditioning. When intensity, color, and spatial position are compressed into a single embedding vector, the model must learn to disentangle these factors internally, often resulting in entangled or imprecise control. By providing distinct tokens for each attribute, the authors give the model explicit axes for manipulation, preventing the interference effects common in latent space conditioning.

This compositional factorization extends beyond lighting. Similar principles apply to material editing, where separating roughness, metallic values, and albedo into distinct tokens could enable precise surface property control. In audio generation, tokenizing pitch, timbre, and tempo separately might yield more instrument-like control over synthetic music. Even in motion generation for video, disentangling camera parameters from object trajectories using tokenized representations could resolve the current ambiguity between moving the camera and moving the subject.

The approach stands in contrast to text-only conditioning, which, while flexible, relies on the model's interpretation of linguistic concepts. Words like "warm" or "dramatic" are inherently subjective and variable. TokenLight replaces this interpretive ambiguity with metric precision; specifying a light source at coordinate (0.5, 1.2, 0.8) with RGB values (255, 200, 150) and intensity 0.8 provides deterministic, reproducible control that creative professionals require for production workflows.

Limitations and Trajectories

Despite its impressive capabilities, TokenLight faces several constraints that warrant consideration. The model's reliance on synthetic training data, while practical for supervision, introduces potential domain generalization issues. Path-traced renders, even high-quality ones, exhibit statistical regularities that differ from real-world photography, particularly regarding sensor noise, lens effects, and complex global illumination phenomena like caustics. While the inclusion of real capture data helps, the synthetic-to-real gap remains an active concern for extreme lighting conditions or unusual material combinations.

Furthermore, the black-box nature of the diffusion model, while beneficial for handling ambiguous cases, lacks the interpretability and editability of explicit physical models. When TokenLight generates a shadow, we cannot query the inferred geometry that cast it. This opacity complicates iterative refinement workflows where artists might need to adjust underlying scene structure rather than just the final pixel output.

The method also assumes a single input view, which inherently limits the consistency of relighting across novel viewpoints. While sufficient for photography editing, extending this approach to 3D-aware, multi-view consistent relighting would likely require integration with neural radiance fields or explicit 3D representations, potentially sacrificing some of the flexibility that makes the current approach so compelling.

Conclusion

TokenLight represents a significant advancement in controllable image generation, demonstrating that precise manipulation of physical phenomena is achievable through careful factorization of conditioning signals. By treating lighting attributes as composable tokens rather than compressed latent variables, the method bridges the gap between the intuitive accessibility of 2D image editing and the precise control traditionally reserved for 3D software.

The work raises fundamental questions about the future of physics-based generation. As diffusion models grow more capable of learning implicit physical constraints from data, we must ask whether the field will move toward hybrid systems that combine neural priors with explicit physical simulations, or whether purely data-driven approaches will suffice for most creative applications. For now, TokenLight offers a practical, immediate solution that advances the state of the art in image relighting while establishing a template for how we might tokenize other continuous physical properties in the generative era. The challenge ahead lies in extending this precision to dynamic scenes, video sequences, and fully 3D environments without sacrificing the interpretability and control that make the approach so powerful.