From Sparse and Imperfect 2D Anchors to Consistent 3D Gaussian Street Scenes: Support-Aware Appearance Distillation | Long Cao et al. | ResearchPod

Transcript: From Sparse and Imperfect 2D Anchors to Consistent 3D Gaussian Street Scenes: Support-Aware Appearance Distillation

Alex: Welcome to another episode of ResearchPod. Today, we're looking at how to turn a few noisy, inconsistent AI-generated images into a perfectly consistent 3D world.

Sam: We're discussing a paper about "appearance baking" for 3D street scenes. The core puzzle is this: how do you take a handful of imperfect images—ones that might have different lighting or strange visual glitches—and use them to update a 3D model so it looks correct from every angle?

Alex: So the goal is something like making a 3D scene look like it's at dusk, even if you only have a few messy reference photos to work from?

Sam: Exactly. And the trap is obvious once you see it. If you just tell the 3D model to copy those photos directly, you copy the mistakes too—the noise, the inconsistencies, all of it gets locked in permanently. So instead, the researchers focus on distilling the change. Not the whole image, just the difference between the original scene and the target look.

Alex: Like applying a photo filter, rather than repainting the whole picture from scratch.

Sam: That's a good way to put it. The technical name for their approach is "teacher-relative appearance residual distillation," but the idea is simpler than it sounds. Think of it like the "track changes" feature in a document editor. Instead of handing someone a brand new document, you just show them what was added, deleted, or moved. The system calculates that difference, then sorts it into layers—broad changes like overall brightness, mid-level changes like colour tone, and fine details like surface texture—so each layer can be handled carefully and separately.

Alex: So the broad "sky turning orange at sunset" change gets handled differently from a tiny, suspicious shadow on the road?

Sam: Precisely. And that separation is what allows the next step to work. They use what they call a "confidence-gated" approach. The system only accepts a change if it's supported by evidence from multiple reference images. If just one photo shows an odd shadow that appears nowhere else, the system treats it as an unreliable outlier and suppresses it. It's a bit like a jury: one witness isn't enough. You need corroboration.

Alex: And all of this is happening inside the 3D model itself?

Sam: Yes. The 3D model they're working with uses a technique called 3D Gaussian Splatting. Imagine representing a whole street scene not as a solid mesh, but as millions of tiny, overlapping blobs—each one storing information about colour, size, and orientation. Crucially, each blob also stores instructions for how its colour should shift depending on the angle you're viewing it from. That's what the "spherical harmonics" are: a compact mathematical recipe for view-dependent colour. The team updates those recipes—that's the "baking" step—to encode the new lighting directly into the scene's structure.

From Sparse and Imperfect 2D Anchors to Consistent 3D Gaussian Street Scenes: Support-Aware Appearance Distillation

Long Cao, Zhongquan Wang, Jie Li, Yuhan Chen, Kefei Qian, Xiangfei Huang, Guofa Li

Jun 24, 2026·5:03·arxiv·Original Paper

3D Gaussian SplattingAppearance BakingTeacher-Relative Appearance Residual DistillationSupport-Aware Residual CoordinationConfidence-Gated Coarse-to-Fine Optimization3D Gaussian Splatting (3DGS)

Listen

5 min

Briefing

Research Question

How can we transform a static 3D Gaussian Splatting (3DGS) street scene to match a target visual condition (e.g., sunset, overcast) when the only available supervision consists of a few sparse, independently edited 2D images? The challenge lies in the fact that these 2D anchors often contain view-specific noise, inconsistent lighting, or hallucinated details that, if fitted directly, would degrade the 3D scene's consistency across novel viewpoints.

Approach

The authors propose a teacher-relative appearance residual distillation framework. Instead of fitting the 3D scene directly to the RGB values of the teacher images, the model calculates residuals—the difference between the teacher's edit and the original render. These residuals are then decomposed into low, mid, and high-frequency bands.

To ensure consistency, the method employs a support-aware lifting mechanism that aggregates these residuals into shared Gaussian primitives only when they are structurally supported and consistent across multiple anchor views. A confidence-gated, coarse-to-fine optimization process then bakes these residuals into the spherical harmonic (SH) coefficients of the Gaussians. This process uses an illumination MLP for broad color shifts, a tone adapter for global statistics, and a detail MLP for local corrections, all while suppressing unsupported high-frequency noise.

Why It Matters

This work enables the creation of reusable, relightable 3D street assets for autonomous driving simulation. By baking appearance changes into the 3D representation itself, the method eliminates the need for expensive, view-dependent image editing or complex inference-time neural decoders. Once the baking process is complete, the teacher model and auxiliary training modules are discarded, allowing the scene to be rendered using a standard, high-speed 3DGS rasterizer.

Key Terms and Definitions

Teacher-Relative Residual — The difference between a teacher-edited 2D image and the original render, used as the primary signal for distilling appearance changes.
Support-Aware Lifting — A process that filters and assigns residual information to 3D Gaussian primitives based on structural, visibility, and cross-anchor consistency.
Spherical Harmonic (SH) Coefficients — The parameters in 3D Gaussian Splatting that represent the view-dependent color and appearance of each primitive.
Confidence-Gated Optimization — A training strategy that selectively admits high-frequency details from the teacher only where they are deemed reliable, preventing the injection of artifacts.
Coarse-to-Fine Baking — An optimization schedule that first establishes global illumination and color statistics before refining local, high-frequency details.

Abstract

Image priors can synthesize target conditions for 3D Gaussian street scenes, but independently edited views do not define a coherent 3D target. Direct fitting can propagate view-specific noise, while existing pipelines do not jointly handle imperfect sparse anchors and standard-rasterizer deployment. To address this gap, teacher-relative appearance residual distillation is introduced for appearance baking. A structured space for frequency decomposition, confidence estimation, and primitive-level lifting is formed by residuals between teacher anchors and original renders. The direct optimization signal is supplied by renderer-space matching, while primitive assignment is regularized by support-aware Gaussian-space aggregation. Supported detail is admitted and unsupported noise is suppressed through confidence-gated coarse-to-fine optimization, after which all residuals are baked into fixed-geometry spherical-harmonic coefficients. The teacher and auxiliary training modules are discarded at inference. Evaluation across Waymo street assets, Tanks and Temples scenes, and multiple target conditions shows a favorable overall balance of target alignment, content preservation, artifact suppression, and cross-view consistency over editing-based baselines. Ablations confirm the effectiveness of the main components. Code will be released at https://github.com/Cagares/Baking-for-3D-Gaussian.

From Sparse and Imperfect 2D Anchors to Consistent 3D Gaussian Street Scenes: Support-Aware Appearance Distillation: Research Paper Podcast