DIFFUSIONBENCH: ON HOLISTIC EVALUATION OF DIFFUSION TRANSFORMERS with a Unified Training Framework Bridging ImageNet and Text-to-Image

The Evaluation Gap in Diffusion Transformers

Diffusion transformer (DiT) research has largely standardized around class-conditional generation on ImageNet as the primary benchmark for progress. While this has enabled rapid iteration, it has created a "benchmark bottleneck" where researchers optimize for ImageNet FID scores without clear evidence that these improvements translate to real-world generative tasks like text-to-image (T2I) generation. This paper investigates whether this narrow focus is sufficient for measuring genuine progress in generative modeling.

Introducing NanoGen

To address the high cost and friction typically associated with multi-task evaluation, the authors introduce NanoGen. This unified training and evaluation framework allows researchers to switch between ImageNet and T2I tasks with minimal configuration changes (approximately 12 lines of code). NanoGen maintains a shared backbone, optimizer, and training loop, ensuring that differences in performance are attributable to the model's capabilities rather than disparate training recipes. The framework supports various diffusion methods, including RAE, VAE, and pixel-space approaches, making it a versatile tool for holistic benchmarking.

Findings: ImageNet vs. T2I

By training 21 latent diffusion models under identical settings, the authors compared performance across both ImageNet and T2I benchmarks. The results show a striking lack of correlation between the two: Pearson correlations between ImageNet FID and common T2I metrics (such as GenEval, DPG-Bench, and GenAIBench) ranged from -0.377 to -0.580. This indicates that a method achieving state-of-the-art results on ImageNet may perform poorly on T2I tasks, suggesting that current research may be inadvertently overfitting to the ImageNet benchmark.

DiffusionBench

Based on these findings, the authors propose "DiffusionBench," a holistic benchmark that requires reporting results on both ImageNet and T2I tasks. By adopting this dual-evaluation approach, the community can better identify methods that reflect broader, more robust progress in generative modeling. The authors provide the code and models to facilitate the adoption of this more rigorous standard.