Inference-Time Scaling of Diffusion Models via Progressive Seed Pruning

Modern generative models like diffusion and flow-matching are highly sensitive to their initial noise seeds. While researchers have explored various ways to spend extra compute at inference time—such as resampling or tree search—most existing methods maintain a constant memory footprint throughout the entire generation process. This paper asks whether relaxing this constraint to allow for variable particle counts can lead to more efficient compute allocation and better final image quality.

The authors introduce Progressive Seed Pruning (PSP), a strategy that treats inference as a search over initial noise seeds. Instead of maintaining a fixed number of samples throughout the denoising process, PSP begins with a large pool of candidate seeds. It uses intermediate denoised estimates—which are already computed as part of the standard denoising process—to score these candidates against a black-box reward function. Based on a predetermined schedule, the algorithm prunes the lowest-performing trajectories, allowing the remaining compute budget to be concentrated on the most promising candidates as they approach the final image.

PSP consistently outperforms standard inference-time scaling baselines, including Best-of-N (BoN), importance-sampling (e.g., FK-Steering), and tree-search methods, when matched for total compute. Across multiple backbones (Stable Diffusion v1.5, SDXL, and SD 3.5), PSP achieves higher scores on automated metrics like GenEval and superior human preference ratings for prompt alignment. Notably, because PSP focuses on seed selection rather than stochastic branching, it remains highly effective even with deterministic solvers, where other resampling-based methods often struggle.

This work demonstrates that inference-time scaling for generative models does not need to be constrained by constant memory usage. By shifting the focus toward early exploration and aggressive pruning, PSP provides a simple, training-free, and deployment-friendly design principle. It allows practitioners to achieve better prompt adherence and higher-quality outputs without requiring additional model training or complex gradient-based guidance.