Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents

Most existing hallucination detection benchmarks focus on natural-language retrieval-augmented generation (RAG). However, modern AI agents increasingly operate over structured inputs like source code, developer-tool outputs, and markdown documents. This paper asks: can we build a unified, span-level benchmark and a corresponding detector that effectively identifies unsupported information across these diverse, structured modalities?

The authors construct a large-scale benchmark containing 74,285 new examples across five sources: SWE-bench (code), Squeez (tool output), ACL paper chunks, READMEs, and Wikipedia markdown. The construction process uses an edit-based injection pipeline: starting with a grounded, correct answer, an LLM injector proposes localized replacements (e.g., changing a function name or a numerical value) to create a hallucination. This method ensures exact character-level labels. To ensure the detector can distinguish between genuine external references and fabrications, the authors implement a reference-grounding step that retrieves relevant library signatures and repository definitions.

The researchers trained a generative detector, LettuceDetect-Qwen-2B, which achieves a span-F1 score of 0.689 on the unified test set. On the challenging code-agent source, it reaches 0.602 span-F1, substantially outperforming both the LettuceDetect-large baseline (0.17) and strong zero-shot LLM judges like Nemotron-3-Ultra (at most 0.22). The model also demonstrates strong performance on established natural-language benchmarks, setting a new state-of-the-art for English PsiloQA IoU (0.724) and remaining highly competitive on RAGTruth.

This work bridges the gap between natural-language RAG verification and the more complex requirements of coding and tool-using agents. By providing a unified, span-level task formulation and a high-quality benchmark, the authors enable the development of more reliable AI systems that can be held accountable for specific, localized errors in code and structured data, rather than just binary accept/reject decisions.