BEYOND PRECISION: IMPORTANCE-AWARE RECALL FOR FACTUALITY EVALUATION IN LONG-FORM LLM GENERATION | Nazanin Jafari et al. | ResearchPod

Transcript: BEYOND PRECISION: IMPORTANCE-AWARE RECALL FOR FACTUALITY EVALUATION IN LONG-FORM LLM GENERATION

Alex: Welcome to another episode of ResearchPod. Today, we are looking at a study that examines a persistent issue in how large language models generate long pieces of text.

Sam: So this paper is basically asking why these models can write a lot of information, but often leave out key details?

Alex: Exactly. The core problem is that while we have spent years teaching artificial intelligence not to lie, we have not been as successful at ensuring it tells the whole truth. And those are genuinely different problems.

Sam: That is a clear distinction. So the research is looking at how we measure whether a model is actually being comprehensive, rather than just accurate?

Alex: Yes. Think of it this way. Most current systems focus on checking if every sentence the model writes is actually true. Researchers call this precision. But this study argues we also need to measure something different — whether the model included all the necessary facts in the first place. That second quality is called recall.

Sam: It is like a student taking an open-book exam. Precision is making sure every answer they write is correct. Recall is checking they did not skip any of the key points the rubric required.

Alex: That is exactly the right way to think about it. And the authors found that current models are much better at the first task than the second. They are fairly good at avoiding false claims, but noticeably worse at ensuring they have covered all the relevant ground.

Sam: How do they actually measure that, though? If a model is writing a long report, how does the system know what counts as missing versus what is simply not relevant?

Alex: They developed a method they call importance-aware recall. The idea is to first build a reference set of facts drawn from trusted sources — essentially a master list of what a complete answer should contain. Then, crucially, they do not treat every fact on that list equally. They assign each fact a weight based on how central it is to the topic. A core fact gets a high weight. A minor supporting detail gets a lower one.

Sam: So it is not just a checklist of facts. It is a system that understands which details are must-haves and which are nice-to-haves.

Alex: Precisely. That weighting is what makes the score meaningful. Without it, a model that omits one critical fact but includes twenty minor ones could still look reasonably complete on paper. With it, the system can tell the difference between a small gap and a genuinely significant omission.

Sam: That does seem like a more honest way to evaluate completeness. But it sounds like it depends heavily on having a good reference set to begin with. What if the answer key itself is incomplete?

BEYOND PRECISION: IMPORTANCE-AWARE RECALL FOR FACTUALITY EVALUATION IN LONG-FORM LLM GENERATION

Nazanin Jafari, James Allan, Mohit Iyyer

Jun 28, 2026·3:12·zotero·Original Paper

Factuality EvaluationImportance-Aware RecallLong-form LLM GenerationPrecision vs. Recall in LLMs

Listen

3 min

Briefing

The Problem with Current Evaluation

Most existing methods for evaluating the factuality of long-form LLM outputs focus almost exclusively on precision. These methods typically decompose a model's response into atomic claims and verify each against external knowledge sources like Wikipedia. While this ensures that what the model says is true, it ignores a critical dimension of factuality: recall. A model might produce a perfectly accurate response that nonetheless omits essential information, leading to factual incompleteness.

A Joint Framework for Factuality

This paper introduces a comprehensive evaluation framework designed to measure both precision and recall simultaneously. By leveraging external knowledge sources, the authors construct a set of reference facts that should ideally be present in a high-quality response. The framework then determines whether these reference facts are captured in the generated text. To make this evaluation more nuanced, the authors incorporate an importance-aware weighting scheme that accounts for the relevance and salience of specific facts, ensuring that missing a critical detail is penalized more heavily than missing a minor one.

Findings on Model Performance

Through their analysis, the authors demonstrate that current LLMs are significantly better at maintaining precision than they are at achieving high recall. This suggests that while models are generally capable of avoiding hallucinations, they suffer from a "completeness" problem. Interestingly, the study finds that models are better at prioritizing and including highly important facts than they are at capturing the full breadth of relevant information, highlighting a specific area where current generation strategies fall short.

Why It Matters

As LLMs are increasingly used for long-form tasks like report generation or summarization, ensuring that they provide complete and comprehensive information is just as important as ensuring they are truthful. This research provides a necessary tool for developers to identify and mitigate the "omission bias" in their models, moving the field toward more reliable and useful AI assistants.

Key Terms and Definitions

Precision — The degree to which the individual claims made by an LLM are factually accurate according to external knowledge sources.
Recall — The extent to which a generated response includes all the relevant and necessary facts required to provide a complete answer.
Atomic Claims — Individual, verifiable statements extracted from a longer text that can be independently checked for truthfulness.
Importance-Aware Weighting — A scoring mechanism that assigns different levels of significance to facts based on their relevance and salience to the overall topic.
Factual Incompleteness — The failure of a model to include all pertinent information in a response, even if the information it does provide is accurate.

Abstract

Evaluating the factuality of long-form output generated by large language models (LLMs) remains challenging, particularly when responses are open-ended and contain many fine-grained factual statements. Existing evaluation methods primarily focus on precision: they decompose a response into atomic claims and verify each claim against external knowledge sources such as Wikipedia. However, this overlooks an equally important dimension of factuality: recall, whether the generated response covers the relevant facts that should be included. We propose a comprehensive factuality evaluation framework that jointly measures precision and recall. Our method leverages external knowledge sources to construct reference facts and determine whether they are captured in generated text. We further introduce an importance-aware weighting scheme based on relevance and salience. Our analysis reveals that current LLMs perform substantially better on precision than on recall, suggesting that factual incompleteness remains a major limitation of long-form generation and that models are generally better at covering highly important facts than the full set of relevant facts.

BEYOND PRECISION: IMPORTANCE-AWARE RECALL FOR FACTUALITY EVALUATION IN LONG-FORM LLM GENERATION: Research Paper Podcast