MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph–Augmented Generation

Retrieval-augmented generation (RAG) has become a standard method for grounding large language models (LLMs) in external knowledge. While traditional RAG focuses on unstructured text, knowledge graph-augmented generation (KG-RAG) has emerged to provide more structured, coherent evidence. However, existing benchmarks for these systems are largely limited to textual data, failing to account for real-world scenarios where information is spread across images, tables, and other non-textual modalities. This paper introduces MKG-RAG-Bench to address this gap, specifically targeting the challenges of multimodal knowledge graph-augmented generation (MKG-RAG).

The authors propose a systematic, cross-domain benchmark built from two multimodal knowledge graphs (general and medical domains). The construction pipeline utilizes an LLM to filter low-utility knowledge, ensuring that the retrieved information is actually relevant to the queries. To create a rigorous evaluation environment, the authors use a controlled masking strategy—hiding either a relation or a tail entity within a triplet—to generate queries that require specific, grounded knowledge to answer. This ensures that the benchmark can isolate the performance of the retrieval component from the generative capabilities of the LLM.

Through experiments with various retriever families (text-only, fusion-based, captioning-based, and reranking-based), the study demonstrates that multimodal retrieval remains a significant challenge. A preliminary analysis shows that naive integration of existing multimodal knowledge graphs can actually degrade performance compared to RAG-free models, often due to the injection of irrelevant or noisy information. By providing a standardized, retrieval-oriented evaluation framework, MKG-RAG-Bench enables researchers to diagnose these failures and develop more effective alignment strategies between multimodal queries and structured graph knowledge.