CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA

Overview

Knowledge-Intensive Visual Question Answering (KI-VQA) requires models to combine visual perception with external world knowledge. While current benchmarks evaluate end-to-end accuracy, they fail to pinpoint whether a model's failure is due to poor visual grounding, incorrect object identification, or flawed reasoning. This paper introduces CRAG-MM-Diagnostics, a diagnostic framework that provides stage-wise annotations to isolate these components, allowing for a granular analysis of where current Vision-Language Models (VLMs) break down.

Research Approach

The authors augment the existing CRAG-MM dataset with new metadata, including target bounding boxes, entity names, and visual complexity scores (e.g., scene crowdedness, target size). They evaluate a range of models—including proprietary VLMs like GPT-5 and open-source models like Qwen—across three stages: language-based visual grounding, object identification, and knowledge retrieval/reasoning. By isolating these stages, the researchers can determine if a model's failure to answer a question is due to an inability to locate the object, an inability to name it, or an inability to retrieve the correct external facts.

Key Findings

The analysis reveals that while visual grounding is a significant hurdle, it is not the primary bottleneck for KI-VQA. Instead, the most frequent point of failure is the knowledge retrieval and reasoning stage. The authors also find that spatial awareness does not emerge naturally in all VLMs; for instance, specialized models like Grounding-DINO often outperform larger, general-purpose VLMs at localizing objects. Furthermore, linguistic ambiguity in user queries significantly degrades performance. By leveraging these insights, the authors propose a grounded bimodal Retrieval-Augmented Generation (RAG) pipeline that crops images based on visual grounding before retrieval, which significantly improves the accuracy of models like GPT-5 and Qwen.

This work shifts the focus of VQA research from "black-box" end-to-end evaluation to modular, diagnostic analysis. By identifying that knowledge retrieval is the primary bottleneck, researchers can prioritize the development of more robust retrieval mechanisms and better-integrated grounding modules. The proposed diagnostic benchmark provides a clear roadmap for building more interpretable and reliable multimodal assistants.