Tarek Al Mustafa, Birgitta König-Ries
6 min
Knowledge graphs (KGs) are increasingly vital for data representation and as memory components for large language models (LLMs). While automated construction methods are proliferating, most research focuses on unstructured text, leaving semi-structured data (such as CSV, JSON, or XML) under-supported. Furthermore, existing evaluation frameworks often lack a holistic approach, failing to connect the quality of generated mappings to the actual usefulness of the resulting KG in downstream applications. This paper addresses this gap by providing a unified, modular benchmark suite.
The authors introduce an evaluation pipeline that assesses KG construction at three distinct stages: (1) mapping predictions, (2) the materialized KG, and (3) downstream task usefulness. The framework is designed to be flexible, allowing systems to output either mapping tables or direct RDF data. It incorporates six quality dimensions: syntactic validity, semantic accuracy, consistency, conciseness, completeness, and pragmatic quality. The latter is measured by a KG's ability to answer a set of expert-curated competency questions (CQs), which serves as a proxy for how well the graph represents the underlying domain knowledge.
The study provides ten expert-curated datasets spanning seven domains, including ecology, genomics, and history. Each dataset includes the base data, target ontologies, gold mappings, and competency questions with gold answers. By providing this resource, the authors enable researchers to perform ablation studies on the impact of optional context (such as ontology tutorials or research papers) and compare systems that employ different strategies for KG construction. The framework also includes a schema-extraction module, allowing systems that produce only RDF to be evaluated at the mapping level.
This work provides a standardized, rigorous way to evaluate automated KG construction, moving the field beyond simple syntactic checks toward assessing whether generated graphs are actually useful for real-world tasks. By isolating the mapping task from the authoring process and providing a clear, multi-stage evaluation, the authors offer a path to better understand error sources in automated pipelines and the degree to which human-in-the-loop intervention is still required.
Knowledge Graphs (KGs) play an increasingly important role in numerous applications ranging from traditional knowledge representation to serving as memory for LLMs to support downstream tasks. However, their construction is labor-intensive; thus, in recent years, numerous approaches for automizing this process have been proposed. Compared to the popularity of construction approaches that focus on textual input data, methods for semi-structured inputs remain underrepresented and as a result, no comprehensive benchmark and evaluation suite exists to judge the quality of mapping predictions and generated KGs. This is problematic, as a KG's quality has direct influence on the downstream applications it supports and thus, strong evaluation mechanisms for their construction are urgently needed. In this work, we thus focus on the evaluation of KG construction from semi-structured data and present a benchmark and evaluation pipeline for KG construction that combines the quality dimensions (1) syntactic validity, (2) semantic accuracy, (3) consistency, (4) conciseness, (5) completeness, and (6) pragmatic quality measured on a KG's ability to provide answers to competency questions. This work contributes a realistic task definition, extends current state of the art evaluation frameworks, allows evaluation of systems that predict mappings and RDF data alike, and combines evaluation of both KG construction and downstream usage. We provide a comprehensive metrics suite, provide ten expert-curated datasets from seven domains, and showcase evaluation using two reference systems.
Alex: [concluding, steady] Precisely. It moves the conversation away from abstract benchmarks and toward the reality of research data, where the cost of a bad mapping is a failed analysis.
Alex: [measured, professional] The most significant finding here is that we can finally evaluate Knowledge Graph construction by its pragmatic utility—specifically, whether the resulting graph can actually answer domain-specific competency questions. This work shifts the focus from mere syntactic correctness to functional performance.
Sam: [curious, analytical] That’s a notable pivot. Usually, we just check if the triples are valid. But you're saying this framework actually tests if the graph is *useful* for a researcher who needs to query their own data?
Alex: [nodding in voice, even pace] Exactly. They argue that a graph can be syntactically perfect and still fail to answer a single research question. By using competency questions as a formal quality dimension, they bridge the gap between building a graph and using it.
Sam: [leaning in, probing] So how does the pipeline handle the disconnect between raw data and the final graph?
Alex: [clear, instructional] It isolates those stages. The framework uses a modular pipeline that evaluates them across three distinct stages: the mapping logic, the materialized graph, and finally, the downstream task performance.
Sam: [thoughtful] That modularity is key. If the mapping is sound but the graph is empty, you know exactly where to look. But how do they handle the "pragmatic quality"—do they just check if the query returns *something*, or are they validating the actual content?
Alex: [precise, measured] They compare the system's query results against gold-standard answers provided by experts. It’s a multi-stage validation process that forces the system to prove it understands the domain, not just the schema.
Sam: [skeptical] That sounds like a high bar. Does the system stay robust if you remove context like ontology documentation, or does the pragmatic quality crash?
Alex: [deliberate] That is the exact question they designed the benchmark to answer. By providing optional context files, they allow researchers to test exactly which inputs contribute most to the final graph's accuracy.
Sam: [reflective] It sounds like a quality assurance assembly line. You check the blueprint, the structure, and finally, whether it’s actually functional. It’s a much more grounded way to measure success.
Alex: [concluding, steady] Precisely.
Alex: [measured, professional] The most significant finding here is that we can finally evaluate Knowledge Graph construction by its pragmatic utility—specifically, whether the graph can actually answer domain-specific competency questions. This shifts the focus from mere syntactic correctness to functional performance.
Sam: [curious, analytical] That’s a notable pivot. Usually, we just check if the triples are valid. But you're saying this framework tests if the graph is actually *useful* for a researcher?
Alex: [nodding in voice, even pace] Exactly. They argue that a graph can be syntactically perfect and still fail to answer a single research question. By using competency questions as a formal quality dimension, they bridge the gap between building a graph and using it.
Sam: [leaning in, probing] So how does the pipeline handle the disconnect between raw data and the final graph?
Alex: [clear, instructional] It uses a modular pipeline that evaluates three distinct stages: the mapping logic, the materialized graph, and the downstream task performance. [[RP_SECTION:validating-pragmatic-quality|Validating Pragmatic Quality]]
Sam: [thoughtful] That modularity is key. If the mapping is sound but the graph is empty, you know exactly where to look. But how do they validate the "pragmatic quality"?
Alex: [precise, measured] They compare the system's query results against gold-standard answers provided by experts. It forces the system to prove it understands the domain, not just the schema. [[RP_SECTION:context-and-robustness-testing|Context and Robustness Testing]]
Sam: [skeptical] That’s a high bar. Does the system stay robust if you remove context like ontology documentation?
Alex: [deliberate] That is the question the benchmark answers. By providing optional context files, they allow researchers to test exactly which inputs contribute most to the final graph's accuracy.
Sam: [reflective] It sounds like a quality assurance assembly line. You check the blueprint, the structure, and finally, whether it’s functional. It’s a much more grounded way to measure success. [[RP_SECTION:limitations-of-competency-questions|Limitations of Competency Questions]]
Alex: [concluding, steady] Precisely. The main limitation is that the pragmatic score depends entirely on the quality of those competency questions.