Intrinsic decomposition and editing of 3D Gaussian splats

How can we perform intrinsic decomposition on 3D Gaussian splatting (3DGS) representations to enable intuitive, physically plausible scene editing? While standard 3DGS captures view-dependent radiance, it entangles material properties and lighting, making it difficult to modify textures without distorting shadows or lighting effects.

The authors propose a pipeline that decomposes a scene into three distinct sets of Gaussian primitives: one for diffuse albedo, one for diffuse shading, and one for view-dependent residuals. By modeling these as independent fields, the system can adapt the distribution and density of Gaussians to the specific characteristics of each layer (e.g., sharp edges for shadows in the shading field vs. high-frequency textures in the albedo field). The reconstruction process uses a video diffusion model to provide temporally consistent albedo priors, regularized by monocular depth estimates to ensure geometric accuracy. Users can then edit the albedo layer by painting on a proxy plane, which is re-projected across multiple views to maintain consistency.

This approach successfully decouples lighting from surface appearance, allowing for high-quality texture and color edits that remain consistent across different viewpoints. Unlike previous methods that often overwrite both texture and shading, this technique preserves the original lighting and shadow structure of the scene. Quantitative evaluations on synthetic and real-world scenes demonstrate that this method achieves superior albedo reconstruction and editing flexibility compared to existing inverse rendering and recoloring tools.

This work provides a practical, real-time-capable solution for editing complex 3D scenes. By separating intrinsic components, it bridges the gap between static 3D reconstruction and interactive content creation, allowing users to modify the appearance of captured environments without requiring manual relighting or complex inverse rendering pipelines.