Nicolas Dufour, Alexei A. Efros, Patrick Pérez
6 min
Abstract
The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed moves FID 3.2x more (in Inception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of the flow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding the FID coefficient of variation (CoV) inside a 1-2% band. (d) Per-cell classifier-free-guidance tuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Based on these findings, we recommend a new FID evaluation protocol: evaluate under per-cell optimal guidance, treat any FID gap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over several training seeds rather than a single FID number.
Alex: Three times more. My instinct would be that using a bigger, more powerful model would smooth that out. Does it?
Sam: That was the researchers' instinct too, and the answer is no. Increasing model size or throwing more computing power at the problem barely changes the spread of results. The study describes this as a "noise floor"—a baseline level of uncertainty that persists regardless of scale. It appears to be a fundamental property of how these models are trained, not something you can simply engineer away.
Alex: So if we can't eliminate the noise, what should the field actually do about it?
Sam: The authors propose a concrete change to how results are reported. Rather than publishing a single number, researchers should run several training experiments and report a range—an error bar that shows how much the score typically varies. If a claimed improvement falls within that range of natural variation, it should be treated as inconclusive. It's the same standard we'd expect in any other area of science.
Alex: It's about being more honest with the data. Did they test any ways to make individual results more consistent?
Sam: They did. They tested an approach called "golden-section guidance tuning." The idea is to systematically search for the best balance point in how the model generates its output—think of it like fine-tuning the contrast knob on a TV until the picture looks sharpest. This does tighten the results. But it comes with two costs: it's computationally expensive, and it can actually change which model appears to win a comparison. So it's not a clean fix.
Alex: So the ranking of models can shift depending on how carefully you tune them. The "best" model really is a moving target.
Sam: It is. And that has real consequences for how the field makes decisions. If a lab sees a small score improvement and concludes they're on the right track, but that improvement is within the noise floor, they may be chasing a signal that isn't there. The study also notes that while their findings were calibrated on specific model architectures, the underlying problem of variability is likely widespread across AI image generation.
Alex: Are there practical costs to adopting stricter reporting standards? Running multiple training experiments takes time and resources.
Sam: It does add cost—more training runs means more time and computing power. But the authors argue the alternative is worse. Publishing results that may be statistically meaningless slows the entire field down, because other researchers build on those results and chase improvements that may not be real. The short-term cost of rigor is lower than the long-term cost of unreliable benchmarks.
Alex: It's a trade-off between speed and accuracy. And right now, the field has been prioritizing speed.
Sam: That's a fair characterization. The study is essentially making the case for a more mature standard of evidence—one where a new model has to prove it's consistently better, not just luckier. It shifts the goal from winning a single lottery draw to demonstrating reliable performance across many draws.
Alex: That framing really clarifies what's at stake. Thanks for walking me through this, Sam.
Sam: My pleasure. This research makes a meaningful contribution to how we think about scientific rigor in AI—a field that moves quickly and doesn't always pause to ask whether its measuring sticks are trustworthy. Thanks for listening to ResearchPod.