LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation | Jennifer A Bishop et al. | ResearchPod

Transcript: LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a challenge in AI: how do you fact-check a long document when your software can only read the first few pages?

Sam: That's the core puzzle. When we use AI to summarize massive reports—like 50-page scientific papers—the software often struggles to stay accurate. Most evaluation tools are what you might call "nearsighted." They have a strict limit on how much text they can process at once, so they simply cut off the end of the document and never check what's there.

Alex: So the AI might produce a summary that looks fine, but it's actually missing facts from the second half of the paper entirely?

Sam: Exactly. And because the checking tool has the same limitation, it can't catch the error either. The authors of this paper set out to fix that. They propose a new framework called LongDocFACTScore—designed to evaluate whether a summary is factually consistent with the entire source text, not just the opening pages.

Alex: How does it actually solve the nearsightedness problem without just overloading the system with more text?

Sam: It uses a smart indexing trick. Think of it like an open-book exam. You don't have to memorize the whole textbook—you just need to know which page to flip to for each question. The system breaks the summary down sentence by sentence, and for each sentence, it scans the full source document to find the specific sections most relevant to that particular claim.

Alex: So instead of reading the whole document from start to finish, it just jumps to the relevant parts?

Sam: Precisely. To do that jumping, it needs a way to measure how similar two pieces of text are in meaning—not just in exact words, but in what they're actually saying. The technique it uses converts each sentence into a kind of coordinate in mathematical space, where sentences with similar meanings end up close together. These are called "sentence embeddings." By comparing those coordinates, the system identifies the top three most relevant passages from the source document for every sentence in the summary.

Alex: And then it checks the summary against those passages?

Sam: Yes. It asks: do these specific passages support what the summary is claiming? It does that for every sentence, then averages the scores to produce one final grade for the whole document. Because it's only pulling small, targeted excerpts rather than feeding in the entire 50-page text at once, the process stays fast and manageable.

Alex: Does the paper show this actually works better than the old approach?

Sam: The authors compared their framework against standard metrics and found it aligned more closely with human judgment. Because it doesn't cut off the document, it catches factual errors buried in sections that nearsighted tools would never reach. It's a meaningful improvement in how we measure AI accuracy on long documents.

LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation

Jennifer A Bishop, Qianqian Xie, Sophia Ananiadou

Jun 28, 2026·5:05·zotero·Original Paper

LongDocFACTScoreFactual ConsistencyLongSciVerifyAbstractive SummarisationReference-free EvaluationSentence Embeddings

Listen

5 min

Briefing

Research Problem

Evaluating the factual consistency of abstractive summaries is a major challenge in natural language processing. While traditional metrics like ROUGE measure word overlap, they fail to capture factual accuracy. More advanced, model-based metrics exist, but they are typically designed for short documents and suffer from restrictive token limits. When applied to long documents, these metrics must truncate the source text, often discarding critical information needed to verify the summary's claims. This study addresses the lack of effective, scalable evaluation methods for long-document summarization.

The LongDocFACTScore Framework

The authors propose LongDocFACTScore, a reference-free evaluation framework designed to scale to documents of any length. Instead of processing the entire source document at once, the framework splits both the summary and the source document into individual sentences. It uses sentence embeddings to identify the most relevant sections of the source document for each summary sentence. By concatenating these relevant snippets, the framework creates a manageable context that can be fed into existing evaluation metrics (such as BARTScore). This approach ensures that the evaluation is both computationally efficient and focused on the specific evidence required to verify each claim in the summary.

Evaluation and Results

To validate the framework, the authors introduced LongSciVerify, a new human-annotated dataset containing scientific and biomedical documents with fine-grained factual consistency labels. They compared LongDocFACTScore against several state-of-the-art metrics, including ROUGE, BERTScore, FactCC, and QuestEval. Experimental results demonstrate that LongDocFACTScore consistently outperforms these baselines, showing a stronger correlation with human judgments of factual consistency across both the LongSciVerify and LongEval datasets. The findings suggest that by intelligently selecting relevant source context, researchers can effectively adapt existing metrics to the complexities of long-document summarization.

Key Terms and Definitions

Abstractive Summarisation — A method of generating a summary by paraphrasing and synthesizing information rather than simply extracting existing sentences from the source.
Factual Consistency — The degree to which a generated summary is supported by or entailed by the information present in the source document.
Reference-free Metric — An evaluation method that assesses a summary by comparing it directly to the source document, rather than requiring a human-written 'gold' summary.
Sentence Embeddings — Numerical vector representations of sentences that capture semantic meaning, used here to calculate similarity between summary and source text.
LongSciVerify — A new dataset created by the authors containing fine-grained human annotations of factual consistency for summaries of long scientific and biomedical documents.

Abstract

Maintaining factual consistency is a critical issue in abstractive text summarisation, however, it cannot be assessed by traditional automatic metrics used for evaluating text summarisation, such as ROUGE scoring. Recent efforts have been devoted to developing improved metrics for measuring factual consistency using pre-trained language models, but these metrics have restrictive token limits, and are therefore not suitable for evaluating long document text summarisation. Moreover, there is limited research and resources available for evaluating whether existing automatic evaluation metrics are fit for purpose when applied in long document settings. In this work, we evaluate the efficacy of automatic metrics for assessing the factual consistency of long document text summarisation. We create a human-annotated data set for evaluating automatic factuality metrics, LongSciVerify, which contains fine-grained factual consistency annotations for long document summaries from the scientific domain. We also propose a new evaluation framework, LongDocFACTScore, which is suitable for evaluating long document summarisation. This framework allows metrics to be efficiently extended to any length document and outperforms existing state-of-the-art metrics in its ability to correlate with human measures of factuality when used to evaluate long document summarisation data sets. We make our code and LongSciVerify data set publicly available: https://github.com/jbshp/LongDocFACTScore.

LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation: Research Paper Podcast