All Relations Lead to Rome: Automated Knowledge Graph Creation and Question Generation

Research Motivation

Modern information retrieval systems, particularly Retrieval-Augmented Generation (RAG), often struggle to balance two distinct paradigms: vector-based retrieval over unstructured text and structured reasoning over knowledge graphs (KGs). While vector-based systems excel at semantic matching, they lack explicit relational structure. Conversely, KG-based systems provide precise reasoning but are often detached from the raw textual evidence. Existing datasets rarely provide a unified resource that aligns these representations with ground-truth entities, relations, and fact-grounded question-answer pairs, limiting the ability to develop and evaluate hybrid retrieval systems.

The ARLtR Framework

To address this, the authors propose All Relations Lead to Rome (ARLtR), a framework designed to jointly construct a knowledge graph and generate fact-grounded question-answer pairs. The process occurs in two main phases:

1. Knowledge Graph Construction: The framework takes a corpus and a weak ontology as input. It segments text into chunks, uses an LLM to extract entities and relations, and integrates them into a structured graph. Finally, it computes embeddings for both the chunks and the entities to support downstream retrieval.
2. Question-Answer Generation: Using predefined configurations (such as question type, complexity, and persona), the system retrieves relevant chunks from the graph. An LLM extracts specific facts from these chunks to serve as ground-truth answers, then generates and reformulates questions based on the desired persona (e.g., novice vs. expert).

Dataset and Utility

The authors instantiate this framework with a historical dataset centered on the Roman Empire, containing over 19,000 entities, 16,000 text chunks, and 8,400 question-answer pairs. By providing a resource where symbolic graph representations are tightly coupled with dense retrieval embeddings, ARLtR enables researchers to evaluate hybrid retrieval strategies and semantic steering techniques that require simultaneous reasoning over relational structure and unstructured semantic content.