CAPEVAL: A DECOUPLED CAPTION EVALUATION ACROSS UNDERSTANDING AND GENERATION

As large-scale image captioning becomes a foundational supervision signal for both multimodal understanding and text-to-image (T2I) generation, it remains unclear how specific properties of these captions—namely, how much information they cover versus how factually accurate they are—differentially influence downstream model performance. This paper introduces CAPEval to systematically disentangle these properties and determine their relative importance for different types of models.

The authors developed CAPEval (Coverage And Precision Evaluation), a benchmark that decomposes caption quality into two distinct metrics: Coverage (the completeness of visual information described) and Precision (the factual reliability of the claims made). The benchmark uses 300 high-resolution images paired with dense, human-verified atomic checklists. To test the impact of these metrics, the researchers conducted controlled end-to-end experiments using 10 different captioning models across four model families. By keeping all other training variables constant, they isolated the effect of caption source on downstream performance for both Vision-Language Models (VLMs) and T2I generators.

The study reveals a clear task-dependent dissociation. For vision-language understanding, regression analysis shows that Coverage is the strongest predictor of success, suggesting that models benefit most from captions that capture a wide breadth of visual details. Conversely, for text-to-image generation, Precision is the only statistically significant predictor, indicating that factual reliability is paramount when captions are used to guide image synthesis. Furthermore, the authors found that captioner scale is not always a proxy for quality; smaller models can outperform larger ones if their output profile—either higher coverage or higher precision—better aligns with the requirements of the downstream task.