TokenLight: Precise Image Relighting with Attribute Tokens and Diffusion Transformers

Forum topic · 小凯 · 2026-04-19

Summary

This post is an in-depth Chinese-language walkthrough of TokenLight (arXiv:2604.15310), a 2026 image relighting framework by researchers from Yale and Adobe Research (Sumit Chaturvedi, Yannick Hold-Geoffroy, Mengwei Ren, et al.). TokenLight enables precise, continuous control of lighting in a single image by representing light as physically interpretable attribute tokens: intensity coefficient, RGB color, ambient ratio, diffusion (softness), global diffusion, 3D position, transition flags, and light-source masks. These tokens are concatenated with image tokens from a VAE latent space and processed jointly by a latent diffusion Transformer with full self-attention, framing relighting as conditional image generation. The model is trained primarily on a large synthetic dataset built from Objaverse 3D assets rendered with path tracing in Blender, complemented by a small set of real photographs to close the domain gap. It achieves state-of-the-art quantitative and qualitative results against Neural Gaffer and DiffusionRenderer on benchmarks with 200 Objaverse objects, and strong user-study ratings. Notable qualitative results include placing lights inside a carved pumpkin, adding convincing backlight to fur, and realistic shadows and caustics near transparent materials, all learned without explicit inverse-rendering supervision. The post also covers limitations (extreme lighting, complex geometry, multi-source scaling) and future directions such as video relighting.

This forum post is an extended Chinese-language interpretation of TokenLight: Precise Lighting Control in Images using Attribute Tokens (arXiv:2604.15310), by Sumit Chaturvedi, Yannick Hold-Geoffroy, and Mengwei Ren (Yale University & Adobe Research, 2026). Below is a structured English summary.

Key points

Problem — image relighting: Given an input image $$I$$ and a lighting change $\Delta L$ , generate $$I_r$$ with identical content but different light: $I_r = f(I, \Delta L)$ . Physically, relighting is hard because of global light transport described by Kajiya's rendering equation (1986): every surface's reflected light becomes another's incident light.
Why prior approaches struggle:
*Inverse rendering* requires recovering geometry, materials, and source parameters from one image — highly ill-posed (e.g., frosted glass, subsurface scattering in petals).
*Appearance learning* faces the curse of dimensionality: intensity, color, position, and softness span a huge continuous space.
Core idea — attribute tokens: TokenLight discretizes lighting control into physically interpretable tokens:

| Token | Meaning | |---|---| | $\lambda$ | light intensity coefficient, $[0,\infty)$ | | $\mathbf{c}=(r,g,b)$ | light color in RGB | | $$a$$ | ambient light ratio | | $$d$$ | diffusion / softness (point light → area soft light) | | $$d_g$$ | global ambient diffusion | | $\mathbf{p}=(x,y,z)$ | 3D light position | | $$t$$ | transition flag between two lighting states | | $\mathbf{m}$ | H×W mask of affected regions |

Architecture: The input image is encoded into latent tokens via a VAE; scalar attributes are encoded with Gaussian Fourier features, vectors directly. Image tokens and lighting-edit tokens are concatenated into one sequence and processed jointly with full self-attention by a latent diffusion Transformer, letting the model reason about spatial content and lighting in the same token space.
Training data: A large synthetic dataset built from Objaverse 3D models rendered with a path tracer in Blender, giving exact ground-truth lighting annotations; plus a small set of real photographs (same scene, toggling actual fixtures) to bridge the domain gap. The synthetic data teaches physical laws; the real data calibrates appearance distribution.

Results

Compared against Neural Gaffer and DiffusionRenderer on 200 random Objaverse test objects (ambient-lit input → point light or environment-map targets), evaluated with PSNR / SSIM / LPIPS: state-of-the-art quantitative and qualitative performance, especially strong on point lights with inverse-square falloff, self-occlusion, and position-dependent penumbras.
User studies confirm human preference for TokenLight outputs.
Trajectory tests and confusion maps show the image response is both sensitive and accurate along continuous paths in lighting-parameter space (measured as a $$B/A$$ ratio of actual vs. ideal change).
Striking qualitative results learned *without explicit inverse-rendering supervision*: placing a light inside a carved pumpkin (occlusion + subsurface glow), adding convincing backlight that rims fur while keeping the face visible, and realistic refraction, caustics, and scattering near glass, ice, and bottles.

Why it works

The physically meaningful tokens act as a strong inductive bias — the model learns a structured, interpretable lighting model rather than a black box.
One unified framework covers three task types: adding spatial virtual lights, editing/diffusing environment illumination, and switching/adjusting in-scene fixtures (up to 3 sources).

Limitations and future work

Degradation on extreme lighting outside the training distribution; imprecise shadows for very complex geometry (dense foliage, fine hair); exponential interaction complexity as light count grows; unreliable for unseen exotic materials.
Promising extensions: temporally consistent video relighting (cf. RelightVid, arXiv:2501.16330; UniRelight, arXiv:2506.15673), interactive editor integration, natural-language-to-token control, and stricter physical constraints (energy conservation, reciprocity).

References cited in the post

Chaturvedi et al. (2026), *TokenLight*, arXiv:2604.15310
Kajiya (1986), *The Rendering Equation*, SIGGRAPH
Debevec et al. (2000), *Acquiring the Reflectance Field of a Human Face*, SIGGRAPH
Chen et al. (2021), *DIB-R++*, NeurIPS
Kocsis et al. (2024), *LightIt*, CVPR
Choi et al. (2025), *ScribbleLight*, CVPR
Futschik et al. (2023), *Controllable Light Diffusion for Portraits*, CVPR
Liang et al. (2025), *DiffusionRenderer*, CVPR
Fang et al. (2025), *RelightVid*, arXiv:2501.16330
He et al. (2025), *UniRelight*, arXiv:2506.15673

*Original post written April 2026, based on arXiv:2604.15310v1.*