Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States

Financial applications often rely on LLMs to process complex filings and tables, where an incorrect answer can lead to significant financial loss. A particularly dangerous failure mode is the confident hallucination, where a model provides a wrong answer with high certainty, bypassing standard uncertainty filters. This paper investigates whether a model's internal activations (the residual stream) contain latent information about answer correctness that is not captured by observable output signals like log-probabilities or self-assessment.

To test this, the author evaluates three instruction-tuned LLMs (Qwen3-8B, Llama-3.1-8B, and Gemma-2-9B) on two financial QA benchmarks, FinQA and TAT-QA. The study defines "confident" answers as those where the model provides the same response across eight independent resamples. The author then trains linear probes on the model's internal residual stream at a fixed depth (2/3 of the network) to predict whether a given answer is correct. These probes are compared against "deployment-cheap" baselines—token log-probabilities and the model's own True/False self-assessment—which are standard in production environments.

The study finds that internal probes are highly effective at identifying errors that remain hidden to output-level metrics. While baseline methods struggle to detect hallucinations among confident answers (achieving 0.55–0.63 AUROC), the linear probes consistently reach 0.68–0.77 AUROC. This advantage is most pronounced in models that tend to over-commit to their answers. The author also establishes that a simple, mean-pooled probe placed at 2/3 of the network depth is a robust design choice that generalizes well across different model architectures, providing a cost-effective way to route suspicious answers to human reviewers.

In high-stakes financial environments, the ability to triage model outputs is critical. By identifying that internal states carry predictive signals for correctness, this research provides a practical, low-compute mechanism for safety monitoring. It demonstrates that even when an LLM appears certain, its internal representations often contain the "truth" of its own error, allowing developers to implement more reliable human-in-the-loop review processes.