Tax authorities and public-sector financial agencies rely on large volumes of unstructured and semi-structured fiscal documents - including tax forms, instructions, publications, and jurisdiction-specific guidance - to support compliance analysis and audit workflows. While recent advances in generative AI and retrieval-augmented generation (RAG) have shown promise for document-centric question answering, existing approaches often lack the transparency, citation fidelity, and conservative behaviour required in high-stakes regulatory domains. This paper presents a multimodal, citation-enforced RAG framework for fiscal document intelligence that prioritises explainability and auditability. The framework adopts a source-first ingestion strategy, preserves page-level provenance, enforces citations during generation, and supports abstention when evidence is insufficient. Evaluation on real IRS and state tax documents demonstrates improved citation fidelity, reduced hallucination, and analyst-usable explanations, illustrating a pathway toward trustworthy AI for tax compliance.
Alex: Welcome to another episode of ResearchPod.
Sam: This paper discusses a system for helping tax experts analyze official documents like IRS forms and guides. The main idea is to use AI that always points back to exact parts of those documents, so answers can be checked and trusted—unlike regular AI that sometimes makes things up.
Alex: So the core problem here is that in tax audits, if the AI guesses wrong about a rule, it could lead to big mistakes or appeals?
Sam: Yes, exactly. Tax agencies deal with piles of PDFs—scanned forms, instructions, policy notes—that vary a lot in layout and change often. Old methods like keyword searches miss the real meaning, and plain AI chat systems invent facts, called hallucinations, which is risky when you need proof from the source.
Alex: Right, so they need AI that's like a careful librarian—pulling quotes with page numbers—instead of just summarizing?
Sam: That's the key. The paper builds on something called retrieval-augmented generation, or RAG. Imagine asking a question: first, the system grabs the most relevant chunks from the documents, like pulling exact sentences or table bits. Then it uses that to generate an answer. But standard RAG can still paraphrase loosely or guess if the match isn't perfect.
Alex: And in tax work, loose paraphrasing isn't enough—you need to trace every claim right back to the page?
Sam: Precisely. This study creates a stricter version: it pulls raw text straight from the PDFs using optical character recognition—or OCR—to read scans without summarizing first, keeping track of which document and page each bit came from. Every answer sentence must cite those sources, or the system regenerates it. If the best match is too weak, it abstains—says there's not enough evidence—instead of risking a wrong answer.
Alex: Huh, that sounds like making AI act more like evidence in court.
Sam: It does. Evaluated on real IRS and state tax docs, it cuts unsupported claims sharply while staying useful for auditors. The paper shows this balances reliability with practicality in high-stakes settings.
Alex: So it was tested on actual IRS forms and state guides—real stuff auditors use. But what made older methods fall short there?
Sam: Traditional setups often relied on fixed rules programmed by hand, like if-then checklists for spotting deductions. Those break easily when rules change or documents vary. Keyword searches miss the deeper meaning, such as when a table implies a condition without spelling it out. Large language models help with natural questions but still invent details or skip sources, which auditors can't verify.
Alex: Right, so plain text search ignores layout, like tables or scans. How do they handle those tricky PDFs without losing track?
Sam: The system starts with a careful pull of raw content—no shortcuts like AI summaries that could twist facts. For scans, it uses OCR, which is like teaching a computer to read printed letters from pictures, turning images back into editable text. Tables get straightened into sentences that repeat headers, so "income over $50,000 gets 20% rate" stays clear. Chunks follow page or section breaks for full context, with a bit of overlap to link ideas.
Alex: That flattening makes sense for checking sources later. Does the real-world testing show it catches what those layout models miss?
Sam: Evaluated on public IRS, California, and New York tax docs—forms, instructions, all multimodal—the approach got high marks from both auto-checks and human reviewers for accurate retrieval and citations. It stands out by skipping fancy visual reasoning at answer time, prioritizing traceable text over complete scene understanding, which fits audit needs. The paper notes this source-first path avoids errors from early abstractions.
Alex: Huh. So it's trading some richness for rock-solid proof links. That's a deliberate choice for tax work.
Alex: But how does it actually pick the right document pieces—without just guessing based on keywords?
Sam: It starts by turning every chunk of text into a list of numbers that captures the overall meaning—like plotting a sentence's ideas on a giant map where similar thoughts cluster close together. Computers compare these number lists by measuring the angle between them; a small angle means the ideas match well. They store these number maps in a fast-search tool, which quickly finds the top few closest matches to your question.
Alex: So it's like finding friends in a crowd by how alike their descriptions are, not exact names. Does that make keyword searches look pretty basic by comparison?
Sam: Yes—keywords count word repeats, missing when ideas connect without the same words. For a question, it grabs the top matches, pulls their text and page info, then feeds only that to a small AI model. This model gets strict rules: use just this evidence, cite every claim with document ID and pages, no outside guesses. After it writes a paragraph, the system checks for those citations; if missing, it tries again up to a few times.
Alex: Right, so failed checks lead to rewriting instead of weak answers. What if even the best match score is too low—does it just stop there?
Sam: Exactly. If the top chunk's match is too weak, it abstains—outputs a clear note that evidence is too thin, rather than risking a loose answer. This ties decisions straight to how well the documents match, catching cases like vague questions or missing topics. In tests on real IRS and state docs, relevant chunks reliably hit strong matches, while others stay low, making the cutoff a solid guardrail.
Alex: Huh. That separation in scores sounds like a meaningful signal for when to trust the system.
Alex: To really judge if it works, they must have tested it on actual questions auditors ask—how did they set that up?
Sam: They created 55 realistic questions based on tax topics like deductions or filing rules, drawn from public IRS forms, California Franchise Tax Board guides, and New York state documents. These mimic what analysts really ask when checking eligibility or income rules, without needing private data or math calculations.
Alex: Right, focused on interpretation from the docs themselves. What metrics did they use to check if citations actually backed the answers?
Sam: Automatic checks first looked at retrieval confidence and whether the system abstained correctly or cited in the right format. For deeper proof, humans reviewed a sample of answers against the cited text, scoring if claims matched evidence, spotting any unsupported bits, judging if abstentions fit weak evidence, and rating overall help for analysts on a 1-to-5 scale.
Alex: Humans verifying makes sense for real reliability. What did those checks show about how well it sticks to the sources?
Sam: In human reviews, cited text fully supported claims in most cases, with unsupported claims or hallucinations at just 1.8%. Abstentions were right about 88% of the time when evidence was thin, and usefulness averaged 4.2 out of 5—clear and practical for audits. This stems from layered checks: similarity gates entry, citations force grounding per paragraph, and humans confirm no loose inferences slip through.
Alex: Huh, so the low hallucination ties directly to those enforced citations and abstentions. Compared to plain setups without them, does it avoid the usual guessing?
Sam: Yes—the paper notes standard RAG without these often paraphrases freely or fills gaps, lacking proof trails that auditors need. Rule-based checklists crack under rule changes, keyword tools miss idea links without word matches, and layout models grasp visuals but falter on reasoning or traceability. Here, source-first text with hard citations builds trust, even if it skips some visual depth.
Alex: That fits the trade-off we discussed—proof over extras. The abstention especially seems key for safe use.
Sam: Precisely. It triggers most on low-match queries lacking clear evidence, forming a defense that keeps outputs conservative and verifiable, as the evaluations confirm.
Alex: So the abstention acts as that final safety net, keeping things conservative when needed. But no system is perfect—what are the main limits they point out in the paper?
Sam: The evaluation sticks to public U.S. IRS and two state tax documents, so it doesn't cover all states or international rules, where formats might differ. OCR on scans can introduce errors, like mangled numbers in tables or cut-off notes. After extraction, everything becomes text, missing chances to reason over visuals or layouts that don't linearize well. It skips numeric calculations or personal data integration, focusing purely on document explanation as support for analysts.
Alex: Right, so it's built for interpretation from public docs, not full tax math or private cases. Those OCR issues make sense for real scans—do they show up often in tests?
Sam: Failures cluster around noisy scans with dense tables or fuzzy text, distorting retrieval slightly, and ambiguous rules across docs that lead to proper abstentions rather than guesses. This conservative stance trades some answer coverage for reliability, fitting tax needs where uncertainty beats error.
Alex: Huh. Prioritizing verifiability over full coverage seems spot-on for audits. How does this stack up against other multimodal setups that summarize tables or images first?
Sam: Summary-based systems embed AI-made abstracts for easier retrieval, but they obscure original sources and add abstraction risks hard to trace later. This source-first text path keeps direct page links, cutting unsupported claims meaningfully—as human checks confirm—without those hidden errors. It favors audit-ready proof over fluency or broader recall.
Alex: That's a clear takeaway for making AI audit-ready—grounding every step in evidence for high-stakes docs. Thanks for breaking it down, Sam—and thanks for listening to this ResearchPod episode on trustworthy fiscal document analysis.