GeoRisk-RAG: A Hierarchy-Aware Risk Framework for Improving RAG Reliability through Selective Answering

This paper investigates how to improve the reliability and factual accuracy of Retrieval-Augmented Generation (RAG) systems in geospatial and natural hazard domains, where conventional semantic similarity metrics often fail to distinguish between topically relevant information and geographically valid information.

To bridge the geographic-validity gap, the authors introduce GeoRisk-RAG, a hierarchy-aware framework that models administrative boundaries and containment relationships as a Directed Acyclic Graph (DAG) derived from Wikidata. The system calculates geographic distance via the Lowest Common Ancestor (LCA) of the query location and retrieved passages, enabling a selective-answering strategy that directly answers exact matches, issues calibrated warnings for broader or narrower contexts, and abstains when contexts belong to different jurisdictions. The evaluation utilizes a newly curated wildfire QA benchmark of 449 manually annotated question-answer pairs spanning multiple administrative granularities.

Experimental evaluations on the wildfire QA benchmark demonstrate that GeoRisk-RAG successfully reduces false confidence rates for location-dependent questions to 0.009, compared to approximately 0.090 for standard semantic similarity and reranking baselines. Furthermore, the framework consistently achieves higher human preference alignment by avoiding the pitfalls of confidently returning topically relevant text that lacks local regulatory validity.

In safety-critical domains such as natural hazard management and emergency response, providing a plausible yet geographically incorrect answer can lead to dangerous real-world decisions. This work establishes a rigorous standard for evaluating end-to-end RAG pipelines by successfully integrating structural spatial topology with selective answering.