Shahaf Valencia Levy, Sagie Benaim
7 min
Abstract
Can a neural network learn to render 3D Gaussian splats on its own, with no rasterizer in the renderer? We present GaussianRenderFormer, a transformer that takes a set of Gaussians and a camera and outputs the image directly. Each Gaussian becomes one token in a view-independent encoder, and a view-dependent decoder turns small patches of camera rays into pixels by attending to those tokens. Trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, GaussianRenderFormer renders 300 unseen objects at 38.08 dB PSNR against the full, unpruned splats, while rasterizing the same pruned input reaches 44.02 dB. The gap is not one of generalization, and our diagnostics are consistent with a bottleneck in the view decoder: telling it the projected footprint of each Gaussian, which it could in principle infer, adds 1.4 dB. Under-training is not ruled out. Code, trained models and videos: https://svlwoof.github.io/gaussianrenderformer
Sam: How does that shared scene representation become one particular picture?
Alex: A decoder, the image-producing stage, starts with small patches of camera rays. Each patch attends to the scene tokens and contributes pixels. The key adaptation is telling that decoder where each Gaussian lands in the image, and how much image area it reaches.
Sam: But the network already receives the Gaussian’s size and orientation, plus the camera. Isn’t that information redundant?
Alex: In principle, yes. The decoder could derive the projected footprint: the blob’s apparent size and shape from that viewpoint. But explicitly supplying it improves performance. The authors also provide projected positions and use smaller image patches to resolve finer detail.
Sam: That sounds close to doing some of the renderer’s work outside the network. Where do they draw the boundary?
Alex: They compute those projections directly from geometry, but nothing is computed per pixel. The network still decides which Gaussians affect a pixel, which are visible, how they blend, and the resulting color. The footprint guides which tokens a patch attends to, rather than directly supplying pixel colors.
Sam: Let’s get to the test. What did they train on, and what counts as the correct image?
Alex: They trained on about twenty-seven thousand objects from Objaverse. Each model input contains twenty thousand Gaussians, pruned from a larger representation. The target is a conventional rendering of that full, unpruned representation.
Sam: Then the network is being asked to reproduce information that pruning might have removed. How do they make the comparison fair?
Alex: They also rasterize exactly the pruned input supplied to the network. Both outputs are compared with the same full-representation reference. That separates the network’s rendering error from the information lost during pruning, although the targets can still contain missing detail.
Sam: And was the final test kept separate from the objects used to choose the design?
Alex: Yes. The main result uses three hundred fresh held-out objects, never used during development. The network reaches about thirty-eight decibels in peak signal-to-noise ratio, an image-error measure where higher is better. Rasterizing the same input reaches about forty-four decibels.
Sam: What does that difference look like, rather than sound like as a score?
Alex: The model generally gets colors and shapes right, while softening edges and fine textures. It can also add high-frequency detail absent from the reference. The metrics are computed around the object, rather than letting a large black background dominate the comparison.
Sam: Could that simply be a generalization problem? Perhaps it renders familiar objects well and struggles with new ones.
Alex: Its training-object scores are only slightly higher than its unseen-object scores. Even on the exact training images and inputs, it remains well below rasterizing those inputs. Comparing its output against its own input’s rasterization also barely changes the score.
Sam: That shifts the question from unfamiliar objects to learning the rendering operation itself. What points specifically toward the decoder?
Alex: Explicitly supplying the projected footprint adds about one-point-four decibels at full scale. That helps both training and held-out objects, despite being information the decoder could theoretically infer. Removing projected positions or returning to larger patches also hurts performance.
Sam: So making the decoder’s image-space job easier helps. Does giving it a more detailed input help too?
Alex: Not when that input contains more Gaussians than it saw during training. In that test, rasterization improves while the network gets worse. That is consistent with difficulty selecting and combining contributions, but it also tests an unfamiliar input size.
Sam: I’d hesitate to call that an architectural bottleneck if the model simply needed more training. Can they rule that out?
Alex: They cannot, and they say so. The final run was still improving when it stopped, and each training image had been seen relatively few times. Several diagnostic experiments were short fine-tunes, not complete retraining. The evidence supports a decoder bottleneck as an interpretation, not a settled diagnosis.
Sam: Even if quality improves, how far away is this from a usable replacement?
Alex: On the tested GPU, one view takes two hundred seventy-five milliseconds. The rasterizer handles the same input in under a millisecond. Reusing the scene encoding helps with additional views, but the network remains much slower and needs far more memory.
Sam: And does it behave like a general renderer when the camera or the scene changes?
Alex: Not yet. It works best in its training camera configuration; changing resolution, distance, or field of view costs quality. Tests on real captures use cropped scene regions viewed from outside, not full scenes with the camera inside. It also uses fixed colors, without view-dependent appearance or relighting.
Sam: Given those limits, who should read the full document, and where should they begin?
Alex: Researchers working on learned rendering or Gaussian representations should start with “The view decoder.” Then read “Main result” and “Where the gap comes from,” especially the matched-input comparison and the training caveats. Read “Limitations” before treating this as a system to deploy.
Sam: And what should everyone else carry away?
Alex: Learning to render these blobs is possible; matching the algorithm it replaces is still the hard part.
Sam: Keep that distinction in mind.