Seemandhar Jain, Keshav Gupta, Manmohan Chandraker
10 min
Abstract
The rapid growth of 3D Gaussian Splatting (3DGS) research demands significant effort to reimplement papers before building on them. We introduce SPLATIFY, a multi-agent framework that converts 3DGS papers into trainable gsplat-based implementations, where generic paper-to-code methods and frontier models fail. SPLATIFY achieves this through five innovations: (1) A context-free grammar for gsplat over a modular method template with extension points for losses, densification, rendering, and optimization, constraining synthesis so generated code satisfies gsplat's architectural invariants by construction. (2) Architectural elements for faithful reproduction: fork-aware citation recovery retrieving component-level code at function-level granularity, Graph-of-Thought synthesis in topological dependency order, RAG-guided in-context example selection from over 20 verified implementations, and visual feedback combining PSNR-guided regeneration, Gaussian-level structural checks, and VLM-driven patching. (3) Knowledge-driven compositional improvement that autonomously finds weaknesses and composes complementary regularizers, losses, and densification strategies to improve upon original results. (4) Interdisciplinary method discovery where agents retrieve physical priors from outside the 3DGS literature and compose them with rendering knowledge to produce methods for previously unaddressed scene types. (5) SPLATIFY-Bench, an evaluation framework across 30 diverse 3DGS papers. On papers without public code, SPLATIFY matches expert implementations while reducing development time from weeks to minutes, and through compositional discovery further improves PSNR by up to 2.4 dB. We additionally demonstrate novel methods for volumetric nebula rendering and other scientific domains, synthesized entirely by SPLATIFY.
Sam: That sounds like giving the model a stable chassis. It can change the parts relevant to the paper without rebuilding every connection.
Alex: For example, a new regularizer goes into the custom-loss hook. A new attribute attached to each blob also needs the relevant creation and splitting hooks. The template preserves the surrounding optimizer and bookkeeping structure, though that doesn’t guarantee the scientific contribution is implemented correctly.
Sam: Can you make the dependency recovery concrete? What does it find that an ordinary citation search might miss?
Alex: The paper uses Trick-GS as an example. It combines techniques for pruning blobs, masking colour components, progressive training, and accelerated rendering. SPLATIFY compares each fork with its parent repository, labels changed functions by their role, and follows inherited changes through the chain.
Sam: So it retrieves the actual modification rather than treating the entire borrowed repository as the contribution. What happens when there’s no code?
Alex: It falls back to extracting the method from the PDF. It also retrieves similar verified implementations as examples. Then a coordinating agent builds files in dependency order, checks interfaces, and runs short training tests.
Sam: A short training test can show that a program runs. It can’t show that the method survives the rest of training.
Alex: That’s why feedback continues beyond the initial test. Agents inspect image errors, look for floating blobs and inconsistent artifacts across views, and propose patches. They also monitor peak signal-to-noise ratio, or PSNR: an image-reconstruction score where higher means less pixel error.
Sam: What does the training curve reveal that looking at an early rendered image would miss?
Alex: Gaussian training has a phase where blobs are added, split, or pruned. A faulty schedule can look acceptable early, then collapse after that phase ends. SPLATIFY checks for late drops and stalled improvement, and targets repairs at the responsible hook.
Sam: Let’s turn to the evidence. How broad was the reproduction test?
Alex: The benchmark contains thirty papers, separated by evaluation purpose. One group has no public code, another has released implementations, and another tests whether distinct technical contributions survive translation. The comparisons use matched training conditions and held-out viewpoints.
Sam: The papers without code seem especially informative. Otherwise, a model might have seen the implementation already.
Alex: That group contains ten papers, paired with commissioned expert implementations. It avoids direct exposure to corresponding public code, which strengthens the paper-driven synthesis test. It doesn’t mean the models lack relevant knowledge of earlier methods.
Sam: How close did the generated implementations get to the experts?
Alex: The authors report average visual quality within half a decibel of expert implementations on PSNR. That’s an average comparison, not a guarantee for every paper. They also compare structural image similarity and perceptual image error.
Sam: And the time saving means what in practice? Are they reporting fast code generation or a complete experimental campaign?
Alex: They report completion in ten to thirty minutes per paper, versus weeks for expert implementations. Full evaluations still involve training on GPUs. I’d treat that as their reported development-time comparison, not a promise that every downstream research campaign finishes that quickly.
Sam: How did the competing systems fare? Some generated code might compile but never produce useful training.
Alex: The authors report that generic paper-to-code systems and frontier models failed to produce stable, paper-level training results in this setup. Their executability breakdown makes distinctions: NERFIFY could produce trainable code, but failed training stability. “Runs” and “reproduces” are different thresholds here.
Sam: There’s another possible shortcut: dropping a difficult contribution while keeping enough of the baseline to score well.
Alex: They test that separately, checking losses, attributes, modules, and schedules against the source papers. An automated judge scores the components, and experienced developers independently inspect them blind to system identity. That supports fidelity beyond aggregate image quality, although it remains rubric-based evaluation.
Sam: Once reproduction works, what does the improvement stage actually search?
Alex: It searches combinations within the same extension points. Loss terms can be added together; competing split-and-prune strategies cannot simply coexist. Explorer agents propose variants, and progressively longer training runs eliminate weak candidates.
Sam: Does that produce a measurable gain, rather than just a collection of alternative implementations?
Alex: The best reported improvement is up to two-point-four decibels over reproduction alone. In one example, agents add patch-structure and geometry-related penalties to DWTGS. That’s a best-case gain, not the average benefit across methods.
Sam: And the new-domain experiments go further: they retrieve physics rather than another rendering technique. What’s the clearest example?
Alex: Nebula rendering. The authors describe nebulae as optically thin emission media, where light adds along the viewing path. Standard transparency compositing can instead produce opaque shells. Agents retrieve radiative-transfer and turbulence ideas, then add emission parameters and penalties addressing opacity, spatial detail, and blob shape.
Sam: Does that mean the system has implemented the correct physical renderer?
Alex: The excerpt supports a narrower conclusion. NebulaGS improves PSNR by about one-point-three decibels over standard Gaussian Splatting, with better cross-view consistency. But it does not change the rasterizer—the machinery projecting blobs into images—so this demonstrates a useful physics-informed adaptation, not comprehensive physical validation.
Sam: That also seems like the boundary on generalization. The system succeeds inside a carefully prepared framework, not an arbitrary software ecosystem.
Alex: That’s my main caution. It uses a specialized template and verified examples. The authors say another domain would require its own grammar and seed implementations. Their ablations also show that removing dependency recovery or training tests substantially weakens results; the infrastructure is doing real work.
Sam: Who should read the full document, and where should they start?
Alex: Rendering researchers should start with Comparison to Expert Implementations, then Novelty Coverage Analysis. Those separate useful images from faithful methods. Researchers building scientific coding agents should read the method template, fork-aware dependency resolution, and ablations before focusing on the invention examples.
Sam: And what should everyone else carry away?
Alex: The strongest lesson is not that a model can write anything from a paper. It’s that a fixed framework, recovered dependencies, and training feedback can make paper-to-code generation reliable in a defined domain.
Sam: Keep that boundary in mind when you read the claims.