Sneha Singhania, Simon Razniewski, Gerhard Weikum
4 min
Extracting structured information from long documents, such as entire books, poses a significant challenge for traditional information extraction (IE) systems. While existing models excel at high-precision extraction from short passages, they often struggle with recall when the relevant entities are scattered across hundreds of pages. This paper addresses the problem of "long-list extraction," where the goal is to identify all entities (objects) that stand in a specific relation to a given subject across a large, unstructured text.
The authors introduce L3X (LM-based Long List eXtraction), a two-stage framework designed to maximize recall while maintaining manageable precision.
Recall-oriented Generation: The system uses a dense retriever to identify a large pool of candidate passages from the document. These passages are re-ranked using heuristics—such as entity mention frequency and pseudo-relevance feedback—and batched to provide the LLM with context-rich inputs. The LLM then generates a comprehensive list of potential object entities.
Precision-oriented Scrutinization: To filter out false positives, the second stage retrieves specific support passages for each candidate object. It then employs various classifiers, including confidence elicitation (asking the LLM to verify the triple) and predicate-specific embedding comparisons, to prune the list and retain only high-confidence extractions.
L3X was evaluated on two new datasets: a collection of 11 fiction books and a large-scale web document corpus. The results demonstrate that L3X significantly outperforms standard LLM-only baselines that rely solely on parametric memory. By optimizing for Recall@PrecisionX (R@Px), the authors show that their method can achieve high recall (80-85%) while maintaining a useful level of precision. This work is significant because it provides a scalable, modular approach to knowledge graph population from long-form content, moving beyond the limitations of single-passage IE.
Sam: Because when a model tries to do both simultaneously, it defaults to what it already "knows" from its training data. Researchers call this its "parametric memory"—essentially, the facts baked into the model during training. The problem is that parametric memory is fixed and incomplete. By forcing the model to search external text and refine that search iteratively, you're moving it from "what do I remember?" to "what can I actually find in this document?" Those are very different questions.
Alex: And the results suggest that distinction matters.
Sam: The paper suggests it does. On complex extraction tasks where standard models typically fall short, L3X achieved recall in the range of 80 to 85 percent. That's a meaningful improvement over approaches that don't use this kind of iterative, document-grounded search.
Alex: Does it work equally well across different types of entities, or are some harder than others?
Sam: That's where the paper is candid about the limits. Some categories—like identifying a character's "opponents"—are genuinely harder, because the textual cues are subtle and context-dependent. A character might never be explicitly called an opponent; you have to infer it from the narrative. L3X struggles more in those cases, which suggests that while the method improves the search process, some facts are just inherently more difficult to extract, regardless of the approach.
Alex: So it's a real improvement, but not a complete solution. The harder the inference required, the more the system still struggles.
Sam: That's a fair summary. What L3X demonstrates is that the way you structure the search matters enormously. By separating the "find everything" phase from the "verify everything" phase, and by letting successful findings guide further searching, you can recover a much more complete picture from a long document. But the underlying difficulty of understanding nuanced language—that remains an open problem.
Alex: It's a useful reminder that even as these systems get more capable, the hardest parts of reading—inference, context, implication—are still genuinely difficult. Thanks for walking us through it.
Sam: It's a paper worth paying attention to, especially as we ask AI systems to do more with longer and more complex documents.
Alex: Thanks for listening to ResearchPod.