DynEval: Holistic Evaluations of T2I Generative Models in the Wild

As text-to-image (T2I) models become more sophisticated, existing automatic evaluation metrics struggle to provide reliable, scalable, and fine-grained feedback. Traditional methods often rely on static prompt sets, expensive human annotations, or complex, resource-intensive pipelines involving proprietary LLMs. This paper asks: can we build a compact, automated evaluator that captures both semantic alignment and image quality while remaining scalable and independent of human labels?

The authors introduce DynEval, a dynamic evaluation framework. To train this system, they first construct GenDB, a large-scale dataset of 500K prompt-image pairs generated by matching prompt complexity with model capability. From this, they derive DynEvalInstruct, a 250K instruction-tuning dataset distilled from a large teacher model (Qwen3-VL-235B). This dataset decomposes evaluation into two streams: text-image alignment (T2IA) via prompt-grounded verification questions, and image quality assessment (IQA) via scene-graph construction and targeted visual reasoning. The authors then fine-tune compact 2B and 4B parameter models using a curriculum learning strategy to distill these evaluation capabilities.

DynEval-2B and DynEval-4B outperform existing automatic evaluators across 11 benchmarks, showing higher correlation with human judgments. Unlike previous methods that often produce contradictory results—such as penalizing high-quality images or rewarding visually distorted ones—DynEval provides a holistic, fine-grained analysis. It successfully identifies failure modes across 42 subcategories and 9 semantic dimensions, offering a scalable alternative to human-in-the-loop evaluation.

This work addresses the "evaluation bottleneck" in generative AI. By providing a compact, open-source evaluator that does not require proprietary LLMs at inference time, it enables researchers to rapidly and reliably benchmark new T2I models. The ability to jointly assess alignment and image quality in a single pass makes it a versatile tool for diagnosing specific model weaknesses in real-world, open-set scenarios.