Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG

Research Context and Problem

Multimodal Retrieval-Augmented Generation (RAG) systems are increasingly used to ground large language models in external evidence. While these systems are typically evaluated on clean data, real-world retrieval often returns topically relevant but unreliable content, such as corrupted metadata, adversarial image patches, or misleading typographic overlays. This paper investigates how such pollution—whether in text or image form—can lead models to generate confident but false answers, a phenomenon the authors term "naive multimodal fusion" failure.

The QIMG-7 Benchmark

To systematically study this, the authors introduce QIMG-7, a controlled stress-test benchmark for multi-sentence factual QA. The benchmark spans four datasets and 16 distinct regimes per question, created by crossing clean/polluted text with clean images or seven different image-attack families (e.g., caption flips, entity swaps, and adversarial patches). This design allows researchers to isolate the specific impacts of text versus image pollution and their interactions.

Source-Aware Trust Resolution (SATR)

Finding that standard multimodal fusion is fragile, the authors propose Source-Aware Trust Resolution (SATR). This training-free approach generates three candidate answers—Parametric (no evidence), Text-only, and Full-MM (text + image)—and then uses an LLM-based resolver to evaluate the reliability of the retrieved sources. The "Field-Selector" variant of SATR uses these reliability assessments to decide whether to trust the multimodal evidence, fall back to text-only, or revert to the model's internal knowledge.

This work demonstrates that unconditional fusion of multimodal evidence is often unsafe in the presence of retrieval pollution. By shifting the paradigm toward "selective trust," the authors provide a practical, training-free framework that significantly outperforms naive approaches. The results suggest that explicit modeling of source reliability is essential for building robust RAG systems capable of handling the messy, adversarial nature of real-world information retrieval.