Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to **23x** rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io
Alex: Preparing a scene more carefully can make every new camera view cheaper to render. The method here learns a compact scene-specific network from an expensive renderer, largely preserving quality while accelerating repeated rendering.
Sam: Faster rendering often means losing fine detail. Is the claim that this avoids that trade-off, or moves the expense somewhere else?
Alex: It moves the expense into scene preparation. This is a method preprint by Chin-Yang Lin and colleagues at NVIDIA and National Yang Ming Chiao Tung University, called Test-Time Baking.
Sam: So the intended user is someone rendering many views of the same scene, rather than requesting a single image?
Alex: That is where the benefit becomes clearest: interactive viewing or rendering a camera trajectory. We'll unpack what the network learns, why earlier shortcuts lost quality, and when the extra preparation actually pays off.
Sam: Start with the task. What information does the system receive, and what is it trying to produce?
Alex: It receives photographs of a scene, together with known camera positions, orientations, and camera settings. The task is novel-view synthesis: rendering that scene from another camera position. This work reconstructs from the supplied observations; it is not a method for freely inventing unseen scenes.
Sam: Many reconstruction systems build a three-dimensional representation first. What is different about the models these authors use?
Alex: Their starting point is the Decoder-only View Synthesis Model, which renders through a neural network without explicitly reconstructing geometry. Its cross-view attention combines information across the input images. That information helps it produce detailed images, but retrieving it repeatedly is expensive.
Sam: Is that repeated retrieval the bottleneck, even after the scene has already been processed?
Alex: Yes. The baseline stores scene features in a key-value cache, a memory that new camera queries consult. More input images mean a larger memory to search through. So adding observations can improve quality while making every subsequent view slower.
Sam: The document compares this with Test-Time Training. What had that approach already solved?
Alex: Test-Time Training updates a small network's weights while processing each scene. Those scene-specific weights then hold information that rendering queries can retrieve. LaCT applies this idea to view synthesis, replacing cross-view attention and making retrieval cost independent of the input-view count.
Sam: That sounds close to the proposal already. Where does the new method depart from it?
Alex: In what the small network learns. LaCT's update learns from raw projected image features, called value tokens. Test-Time Baking instead learns from the outputs of cross-view attention, after information has already been combined across views.
Sam: So attention does the expensive integration first, and the small network learns its answers. Attention becomes a teacher during scene preparation, rather than something each rendering query must use.
Alex: That's the central change. The authors call it collaboration rather than replacement. Their analysis suggests that learning attention outputs spreads the baked network's dependence across more context images, instead of concentrating as strongly on the most influential image.
Sam: But a new camera position has no photograph attached. How can a network trained on image features know what to do with that?
Alex: That is the other central contribution. Conventional preparation feeds the small network features containing both camera information and image content. Rendering feeds it camera information without the image. The authors identify that mismatch as an unnecessary generalization problem.
Sam: How do they remove the mismatch without throwing away the photographs that describe the scene?
Alex: They make a pair of feature streams for each input view. One contains the photograph and camera information; it supplies the attention-produced learning target. The other contains camera information only; it supplies the small network's input.
Sam: Then the network practises retrieving scene information without seeing the photograph in its input. That matches what it must do for a new camera query.
Alex: And that matching continues through the network's layers, not just at the beginning. The baked network processes both the camera-only preparation stream and the novel-view stream. Once preparation is complete, rendering uses the baked weights instead of consulting the large scene cache.
Sam: Give me a concrete sense of what survives this compression. What details do the authors actually show?
Alex: In their scene comparisons, they point to handrails, window grilles, and small text. Test-Time Baking preserves thin structures and fine detail that LaCT and the earlier Large View Synthesis Model render with visible blur. That is qualitative evidence, alongside numerical image-quality comparisons.
Sam: Let's separate the headline from the conditions. How substantial is the evaluation?
Alex: The main evaluation uses 140 held-out scenes from the DL3DV dataset. It tests sparse-to-denser collections of input views at the paper's high-resolution setting. The strongest reported rendering speedup occurs with 64 input views.
Sam: What does that speedup mean in actual playback terms?
Alex: The proposed method renders at 23 frames per second, compared with about one for the attention baseline in that dense-view setting. That's the headline, but quality needs a qualifier: the standard baked model still trails the baseline on several image-quality measures.
Sam: Then “comparable quality” is not a claim that the compression is lossless. How do they address the remaining gap?
Alex: A fixed-size network eventually runs out of capacity as more images arrive. Their inference-time workaround groups nearby camera views and bakes a separate network copy for each group. A new camera query uses the copy belonging to its nearest group.
Sam: That keeps each rendering query local, but the total stored scene state must grow. Does it actually recover quality?
Alex: In the dense-view benchmark, it brings quality much closer to the attention baseline, with the same reported rendering speed. Some scores slightly exceed that baseline, while others remain slightly worse. The routing is heuristic, and the copies increase storage, so this is not unlimited capacity for free.
Sam: How much evidence says the proposed ingredients matter, rather than just inheriting a strong pretrained model?
Alex: Their component experiments separately test camera-only pairing, the attention-derived target, and the inner-loop fitting procedure. Each improves quality in the controlled setup. Training from scratch still beats the pretrained basic Test-Time Training replacement, which supports a structural contribution beyond initialization.
Sam: And the faster comparison with LaCT does not isolate the baking idea alone, does it?
Alex: It doesn't. The proposed model also has a shallower decoder than LaCT, so that speed difference reflects architecture too. Beyond the main dataset, evaluations on outdoor and other out-of-domain scenes generally support the quality-speed trade-off, but do not establish universal parity.
Sam: The remaining question is waiting time. When does slower scene preparation stop cancelling the rendering gain?
Alex: Using their reported timings, full baking overtakes the attention baseline after roughly five to seven rendered views. For a single requested image, the extra preparation can be a drawback. And “constant rendering cost” means independent of context size, not independent of image resolution or all other computation.
Sam: Who should read the full document, and where should they start?
Alex: Researchers building neural view synthesis systems should read the sections on repurposing Test-Time Training and correcting the input mismatch. Then examine the component ablations and the end-to-end efficiency analysis. Those show both why the method works and whether its preparation cost fits your workload.
Sam: And what should everyone else remember?
Alex: Expensive attention can teach a compact scene network before rendering begins. The gain comes from paying once to integrate the observations, then reusing what was learned—with capacity limits still attached.
Sam: Keep both the preparation bill and the reuse in view.