TriQua: Reconciling Granularity and Context in Factuality Evaluation

Factuality evaluation for Large Language Models (LLMs) typically follows a 'decompose-then-verify' paradigm. However, this approach faces a persistent trade-off: decomposing text into strictly atomic facts (e.g., simple subject-relation-object triples) often strips away essential context, while keeping broader statements intact makes it difficult to pinpoint exactly which part of a claim is incorrect. TriQua addresses this by adopting a hyperrelational structure. It extracts simple claims as standard triples and complex claims as base triples augmented with independent 'qualifiers' (e.g., temporal or spatial constraints). This allows the system to maintain the atomicity of the core assertion while explicitly attaching the necessary context required for accurate verification.

The TriQua framework consists of three main stages: decomposition, retrieval, and verification. During decomposition, the system uses few-shot prompting to extract facts into the structured format. For retrieval, these structured units are converted into query strings that include both the base triple and its qualifiers, ensuring the retriever has the full context. The verification step uses an LLM to evaluate these units against retrieved evidence. Crucially, when a fact is found to be unsupported, the verifier is instructed to annotate the specific component—either the base triple or a specific qualifier—that is incorrect. This provides fine-grained explainability that previous, less structured methods lack.

To quantify performance, the authors introduce TriQuaScore. Unlike traditional metrics that treat an entire claim as a single unit, TriQuaScore treats the base triple and each individual qualifier as separate, quantifiable 'fact units.' By calculating the ratio of supported fact units to the total number of units, the metric provides a more precise and diagnostic assessment of factuality. The framework also includes a deduplication step to ensure that repeated contextual information does not artificially inflate or deflate the final score. Empirical results across multiple backbone models show that TriQuaScore aligns strongly with human-annotated factuality and consistently outperforms existing decomposition-based frameworks in error reduction and diagnostic accuracy.