WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization

Overview

WeGenBench is a comprehensive diagnostic benchmark designed to evaluate text-to-image (T2I) generation models across multiple dimensions. While existing benchmarks often focus on broad semantic coverage or single-scenario tasks, they frequently fail to provide a granular, interpretable diagnosis of model weaknesses. WeGenBench addresses this by offering a bilingual (Chinese and English) dataset of 4,000 prompts, systematically categorized into 16 macroscopic scenarios and annotated with fine-grained capability tags.

Methodology

The benchmark employs a dual-layer taxonomy. The first layer classifies prompts into 16 major scenarios (e.g., Portraits, Food, Architecture, Logos), while the second layer uses a pool of capability tags (e.g., Spatial, Negation, Attribute Binding) to define specific generative challenges. This structure allows researchers to move beyond holistic scoring and identify exactly where a model fails—such as struggling with complex spatial relationships or specific linguistic nuances in Chinese versus English.

Evaluation Framework

To ensure the evaluation is both accurate and interpretable, the authors integrate Vision-Language Models (VLMs) to provide automated, diagnostic feedback. The framework assesses performance across three core dimensions:

1. Semantic Alignment: Uses both Question Answering (QA) and Chain-of-Thought (CoT) deduction to verify consistency.
2. Aesthetic Quality: Employs an anchor-based grading method to maintain consistency while reducing computational costs.
3. Visual Text Rendering: Combines OCR and VLM analysis to check for spelling and legibility.

Crucially, this system generates explicit rationales for its scores, allowing users to verify the reasoning behind the assessment rather than treating the evaluation as a black box.