The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed moves FID 3.2x more (in Inception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of the flow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding the FID coefficient of variation (CoV) inside a 1-2% band. (d) Per-cell classifier-free-guidance tuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Based on these findings, we recommend a new FID evaluation protocol: evaluate under per-cell optimal guidance, treat any FID gap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over several training seeds rather than a single FID number.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at how researchers measure progress in AI image generation—and whether those measurements are as reliable as we think. Sam, what's the central puzzle here?
Sam: We're looking at a study focused on something called the Fréchet Inception Distance, or FID. Think of FID as a report card score for AI-generated images—it's the standard way researchers judge whether a new model produces more realistic pictures than the last one. The study's core finding is that because of random factors baked into the training process, the single number researchers typically report may be misleading.
Alex: So the paper is asking whether we're measuring genuine scientific progress, or just getting lucky?
Sam: Exactly. Here's an analogy. Imagine you're testing a new recipe, but every time you cook it, you use a slightly different oven temperature, add ingredients in a different order, and throw in a random pinch of salt. You'd expect the dish to taste a little different each time—even if the recipe itself hasn't changed. Training an AI model works the same way. The researchers found that if you train the exact same model twice, the FID score changes noticeably—not because the model improved, but because of this built-in randomness.
Alex: That sounds like a serious problem. If the score shifts just from retraining, how can anyone trust a claim that their new model is the best?
Sam: That's precisely the issue. A researcher might publish a result showing their model improved the score by a small margin. But this study suggests that gap could simply be a lucky training run—what the authors call the "training lottery"—rather than a genuinely better algorithm.
Alex: Can you walk me through where that randomness actually comes from?
Sam: Sure. There are three main sources. First, there's initialization—before training even begins, the model's internal settings are assigned random starting values, like picking random numbers to start a board game. Second, there's data ordering—the model learns by looking at millions of images, and the sequence in which it sees them changes every run. Third, modern AI models deliberately inject small amounts of random noise throughout training to keep the learning process flexible and prevent it from getting stuck. Any one of these would introduce some variation. Together, they mean the final model is just one of many possible outcomes from the same starting recipe.
Alex: So the model we end up with is essentially the result of a lottery—one draw from a huge range of possible models?
Sam: That's a good way to put it. And the study quantifies just how large that lottery effect is. Retraining a model from scratch produces roughly three times more variation in the final FID score than simply generating a new batch of images from the same trained model. The randomness in training is far larger than the randomness in the output.
Alex: Three times more. My instinct would be that using a bigger, more powerful model would smooth that out. Does it?
Sam: That was the researchers' instinct too, and the answer is no. Increasing model size or throwing more computing power at the problem barely changes the spread of results. The study describes this as a "noise floor"—a baseline level of uncertainty that persists regardless of scale. It appears to be a fundamental property of how these models are trained, not something you can simply engineer away.
Alex: So if we can't eliminate the noise, what should the field actually do about it?
Sam: The authors propose a concrete change to how results are reported. Rather than publishing a single number, researchers should run several training experiments and report a range—an error bar that shows how much the score typically varies. If a claimed improvement falls within that range of natural variation, it should be treated as inconclusive. It's the same standard we'd expect in any other area of science.
Alex: It's about being more honest with the data. Did they test any ways to make individual results more consistent?
Sam: They did. They tested an approach called "golden-section guidance tuning." The idea is to systematically search for the best balance point in how the model generates its output—think of it like fine-tuning the contrast knob on a TV until the picture looks sharpest. This does tighten the results. But it comes with two costs: it's computationally expensive, and it can actually change which model appears to win a comparison. So it's not a clean fix.
Alex: So the ranking of models can shift depending on how carefully you tune them. The "best" model really is a moving target.
Sam: It is. And that has real consequences for how the field makes decisions. If a lab sees a small score improvement and concludes they're on the right track, but that improvement is within the noise floor, they may be chasing a signal that isn't there. The study also notes that while their findings were calibrated on specific model architectures, the underlying problem of variability is likely widespread across AI image generation.
Alex: Are there practical costs to adopting stricter reporting standards? Running multiple training experiments takes time and resources.
Sam: It does add cost—more training runs means more time and computing power. But the authors argue the alternative is worse. Publishing results that may be statistically meaningless slows the entire field down, because other researchers build on those results and chase improvements that may not be real. The short-term cost of rigor is lower than the long-term cost of unreliable benchmarks.
Alex: It's a trade-off between speed and accuracy. And right now, the field has been prioritizing speed.
Sam: That's a fair characterization. The study is essentially making the case for a more mature standard of evidence—one where a new model has to prove it's consistently better, not just luckier. It shifts the goal from winning a single lottery draw to demonstrating reliable performance across many draws.
Alex: That framing really clarifies what's at stake. Thanks for walking me through this, Sam.
Sam: My pleasure. This research makes a meaningful contribution to how we think about scientific rigor in AI—a field that moves quickly and doesn't always pause to ask whether its measuring sticks are trustworthy. Thanks for listening to ResearchPod.