Can a neural network learn to render 3D Gaussian splats on its own, with no rasterizer in the renderer? We present GaussianRenderFormer, a transformer that takes a set of Gaussians and a camera and outputs the image directly. Each Gaussian becomes one token in a view-independent encoder, and a view-dependent decoder turns small patches of camera rays into pixels by attending to those tokens. Trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, GaussianRenderFormer renders 300 unseen objects at 38.08 dB PSNR against the full, unpruned splats, while rasterizing the same pruned input reaches 44.02 dB. The gap is not one of generalization, and our diagnostics are consistent with a bottleneck in the view decoder: telling it the projected footprint of each Gaussian, which it could in principle infer, adds 1.4 dB. Under-training is not ruled out. Code, trained models and videos: https://svlwoof.github.io/gaussianrenderformer
Alex: A neural network can draw an object from transparent blobs, but the standard algorithm still wins. This study replaces that drawing algorithm with a learned model, then asks why it falls short.
Sam: So the contribution is showing that this can work, rather than making rendering faster or better?
Alex: That’s the distinction. GaussianRenderFormer is a method paper and arXiv preprint by Shahaf Valencia Levy and Sagie Benaim, at the Hebrew University of Jerusalem. Their network produces images of objects it never saw during training, but it remains slower and less accurate than the conventional renderer.
Sam: Who should care about that intermediate result, if the existing algorithm still does the job better?
Alex: Researchers trying to build learned renderers that could eventually handle richer lighting or combine different scene representations. Those are motivations, not achievements here. We’ll unpack what the network learns, the comparison that exposes its limits, and why the authors suspect the image-producing stage.
Sam: Start with the representation. What are these blobs, and what does the usual renderer do with them?
Alex: They’re called three-dimensional Gaussian splats: overlapping, partly transparent blobs that represent an object or scene. Each has a position, size, orientation, color, and opacity. A rasterizer, the usual image-making algorithm, projects them onto the image and blends their contributions.
Sam: That sounds like more than placing colored dots. The renderer has to decide what shows through what.
Alex: Yes. These blobs overlap heavily, without the explicit surfaces that triangle meshes provide. The standard renderer sorts them to work out their contributions. A learned renderer must instead resolve visibility and blending from the input.
Sam: What had already been learned successfully, and what changes here?
Alex: An earlier model called RenderFormer learned to render triangle meshes using a transformer, a network that combines information through attention. Its successor extended that approach to more scene ingredients. According to these authors, existing Gaussian methods still handed final image production to a handwritten renderer.
Sam: So they borrow a mesh renderer, but ask it to handle a much larger collection of overlapping, transparent elements.
Alex: They start from RenderFormer’s pretrained weights. Each Gaussian becomes a token, meaning a numerical representation the network can process. A scene encoder lets those tokens exchange information, producing a representation that can be reused across camera views.
Sam: How does that shared scene representation become one particular picture?
Alex: A decoder, the image-producing stage, starts with small patches of camera rays. Each patch attends to the scene tokens and contributes pixels. The key adaptation is telling that decoder where each Gaussian lands in the image, and how much image area it reaches.
Sam: But the network already receives the Gaussian’s size and orientation, plus the camera. Isn’t that information redundant?
Alex: In principle, yes. The decoder could derive the projected footprint: the blob’s apparent size and shape from that viewpoint. But explicitly supplying it improves performance. The authors also provide projected positions and use smaller image patches to resolve finer detail.
Sam: That sounds close to doing some of the renderer’s work outside the network. Where do they draw the boundary?
Alex: They compute those projections directly from geometry, but nothing is computed per pixel. The network still decides which Gaussians affect a pixel, which are visible, how they blend, and the resulting color. The footprint guides which tokens a patch attends to, rather than directly supplying pixel colors.
Sam: Let’s get to the test. What did they train on, and what counts as the correct image?
Alex: They trained on about twenty-seven thousand objects from Objaverse. Each model input contains twenty thousand Gaussians, pruned from a larger representation. The target is a conventional rendering of that full, unpruned representation.
Sam: Then the network is being asked to reproduce information that pruning might have removed. How do they make the comparison fair?
Alex: They also rasterize exactly the pruned input supplied to the network. Both outputs are compared with the same full-representation reference. That separates the network’s rendering error from the information lost during pruning, although the targets can still contain missing detail.
Sam: And was the final test kept separate from the objects used to choose the design?
Alex: Yes. The main result uses three hundred fresh held-out objects, never used during development. The network reaches about thirty-eight decibels in peak signal-to-noise ratio, an image-error measure where higher is better. Rasterizing the same input reaches about forty-four decibels.
Sam: What does that difference look like, rather than sound like as a score?
Alex: The model generally gets colors and shapes right, while softening edges and fine textures. It can also add high-frequency detail absent from the reference. The metrics are computed around the object, rather than letting a large black background dominate the comparison.
Sam: Could that simply be a generalization problem? Perhaps it renders familiar objects well and struggles with new ones.
Alex: Its training-object scores are only slightly higher than its unseen-object scores. Even on the exact training images and inputs, it remains well below rasterizing those inputs. Comparing its output against its own input’s rasterization also barely changes the score.
Sam: That shifts the question from unfamiliar objects to learning the rendering operation itself. What points specifically toward the decoder?
Alex: Explicitly supplying the projected footprint adds about one-point-four decibels at full scale. That helps both training and held-out objects, despite being information the decoder could theoretically infer. Removing projected positions or returning to larger patches also hurts performance.
Sam: So making the decoder’s image-space job easier helps. Does giving it a more detailed input help too?
Alex: Not when that input contains more Gaussians than it saw during training. In that test, rasterization improves while the network gets worse. That is consistent with difficulty selecting and combining contributions, but it also tests an unfamiliar input size.
Sam: I’d hesitate to call that an architectural bottleneck if the model simply needed more training. Can they rule that out?
Alex: They cannot, and they say so. The final run was still improving when it stopped, and each training image had been seen relatively few times. Several diagnostic experiments were short fine-tunes, not complete retraining. The evidence supports a decoder bottleneck as an interpretation, not a settled diagnosis.
Sam: Even if quality improves, how far away is this from a usable replacement?
Alex: On the tested GPU, one view takes two hundred seventy-five milliseconds. The rasterizer handles the same input in under a millisecond. Reusing the scene encoding helps with additional views, but the network remains much slower and needs far more memory.
Sam: And does it behave like a general renderer when the camera or the scene changes?
Alex: Not yet. It works best in its training camera configuration; changing resolution, distance, or field of view costs quality. Tests on real captures use cropped scene regions viewed from outside, not full scenes with the camera inside. It also uses fixed colors, without view-dependent appearance or relighting.
Sam: Given those limits, who should read the full document, and where should they begin?
Alex: Researchers working on learned rendering or Gaussian representations should start with “The view decoder.” Then read “Main result” and “Where the gap comes from,” especially the matched-input comparison and the training caveats. Read “Limitations” before treating this as a system to deploy.
Sam: And what should everyone else carry away?
Alex: Learning to render these blobs is possible; matching the algorithm it replaces is still the hard part.
Sam: Keep that distinction in mind.