Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, Benjamin Van Durme
5 min
As generated text becomes more commonplace, it is increasingly important to evaluate how well-supported such text is by external knowledge sources. Many approaches for evaluating textual support rely on some method for decomposing text into its individual subclaims which are scored against a trusted reference. We investigate how various methods of claim decomposition -- especially LLM-based methods -- affect the result of an evaluation approach such as the recently proposed FActScore, finding that it is sensitive to the decomposition method used. This sensitivity arises because such metrics attribute overall textual support to the model that generated the text even though error can also come from the metric's decomposition step. To measure decomposition quality, we introduce an adaptation of FActScore, which we call DecompScore. We then propose an LLM-based approach to generating decompositions inspired by Bertrand Russell's theory of logical atomism and neo-Davidsonian semantics and demonstrate its improved decomposition quality over previous methods.
As automated evaluation of generated text becomes standard, researchers increasingly rely on decomposing long claims into smaller, atomic subclaims to verify factual accuracy. This paper investigates whether the choice of decomposition method—often treated as a black-box step—significantly influences the final evaluation scores of models. The authors ask: how do different decomposition strategies affect downstream metrics, and can we develop a more rigorous, theoretically grounded approach to this process?
The authors evaluate several existing LLM-based and parsing-based decomposition methods. They demonstrate that downstream metrics like FActScore are not independent of the decomposition method used, meaning that differences in reported factual precision may stem from the decomposition strategy rather than the model being evaluated. To address this, they introduce DecompScore, a metric that measures the quality of a decomposition by counting how many subclaims are supported by the original source text. Finally, they propose a new decomposition strategy inspired by Bertrand Russell’s theory of logical atomism and neo-Davidsonian semantics, which breaks sentences down into the smallest possible units of meaning (unary and binary predicates).
The study finds that FActScore values vary significantly depending on the decomposition method, confirming that the decomposition step is a major source of measurement error. Among the tested methods, the authors' proposed approach (R-ND) consistently achieves the highest DecompScore, indicating that it produces more atomic and coherent subclaims than previous methods. This suggests that grounding decomposition in formal semantic theory leads to more reliable and granular evaluation of generated text.
This work highlights a critical vulnerability in current evaluation pipelines for large language models. By showing that the "fact-checking" process is sensitive to how claims are parsed, the authors provide a necessary framework for researchers to audit their evaluation metrics. The introduction of DecompScore and the R-ND method offers a path toward more consistent, transparent, and precise evaluation of model-generated content.
Sam: So instead of keeping a tangled sentence together, you pull it apart into the simplest building blocks—like separating the ingredients of a recipe rather than judging the whole dish at once.
Alex: Exactly. They use this as a blueprint for how to instruct the AI to do its splitting. The specific technique they use is called "Neo-Davidsonian" prompting—which is a technical label for a very practical idea. It means they give the AI a structured set of instructions that forces it to identify every single relationship in a sentence: who did what, when, where, and how. Nothing gets bundled together that should be separate.
Sam: So the philosophy gives them a principled method—a kind of grammar for facts—that makes the splitting more precise and consistent.
Alex: That's the central claim. And the reason it matters is this: when the splitting is sloppy, we end up blaming the AI for errors that were actually introduced by our own measurement process. Better structure in the breakdown means we can finally see whether the AI itself is the source of a mistake, or whether it was the tool we used to evaluate it.
Sam: But here's something I want to push back on. If DECOMPSCORE filters out the pieces that don't match the original sentence, aren't we just hiding the AI's tendency to invent facts—what people call "hallucination"?
Alex: It's a fair challenge, and the answer is that it works the other way around. The pieces being filtered out aren't the AI's invented facts—they're fragments that the splitting tool accidentally introduced. The AI's actual claims stay in. What gets removed is the noise created by a clumsy breakdown. So rather than concealing hallucination, the filter clears away the interference so you can see the AI's real output more clearly.
Sam: So it's about fairness in evaluation. You need to be certain the error belongs to the AI, not to the tool measuring it.
Alex: Precisely. And the authors are careful not to overclaim. They acknowledge that even with this improved method, some sentences are genuinely difficult to break down cleanly—particularly ones with complex logical structures or ambiguous references. They present this as a meaningful step forward, not a complete solution.
Sam: There's something genuinely humbling about that. We invest enormous effort in building more capable AI, but if the ruler we're using to measure it is warped, we may never get an accurate reading.
Alex: That's a fair way to close on it. The paper's underlying argument is that measuring truth isn't just about having good AI—it's about having good tools for asking the question in the first place. Before you can reliably check whether an AI is telling the truth, you have to master the art of breaking language down into its most basic, honest components. Thanks for listening to ResearchPod.