Video Generation Models Are Inherent Lighting Estimators

Recovering dynamic environment maps from a single in-the-wild video is essential for photorealistic rendering and virtual object insertion, yet it remains a significant challenge due to complex camera motion, occlusions, and the difficulty of capturing high dynamic range (HDR) data. The authors investigate whether modern video generation models, which exhibit an implicit understanding of physical lighting, can be repurposed as effective lighting estimators.

The authors introduce V-LITE, a framework that reformulates lighting estimation as a video inpainting task. Inspired by visual effects (VFX) industry practices, the model is tasked with inserting a synthetic chrome ball into a video; the reflections captured on this ball serve as a proxy for the scene's dynamic illumination. To adapt existing LDR-native video diffusion models for this task, the authors design an HDR-aware Variational AutoEncoder (VAE) with learnable tonemap adapters and employ a LoRA-based fine-tuning strategy. They also construct V-LITESet, a hybrid dataset containing over 8K HDR video sequences and 800 static HDR images, to provide both dynamic temporal context and absolute photometric anchors.

V-LITE demonstrates that modern video diffusion models are inherently capable of estimating complex, physically plausible lighting. By treating the problem as an inpainting task rather than a simple translation, the model achieves superior temporal consistency compared to per-frame image-based lighting estimation methods. The framework successfully generates high-fidelity HDR environment maps that allow for the seamless insertion of virtual objects into real-world footage, with lighting and shadows that remain consistent even under rapid camera movement.

This work provides a robust, data-driven solution for dynamic HDR lighting estimation that does not require specialized multi-exposure hardware or controlled capture environments. By unlocking the internal generative priors of video diffusion models, V-LITE offers a scalable approach to photorealistic augmented reality and scene relighting, significantly reducing the computational and logistical barriers to high-quality environment map recovery.