From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection

Automatic Speech Recognition (ASR) models, particularly large-scale ones like Whisper, are prone to hallucinations—fluent, plausible-sounding transcriptions that lack any basis in the actual audio input. These errors pose significant risks to downstream NLP applications. This paper investigates whether these hallucinations can be reliably detected without relying on ground-truth reference transcripts, comparing three distinct paradigms: text-based metrics, LLM-based analysis, and probing the internal decoder states of the Whisper model.

The authors evaluated detection performance using the HALAS dataset, which contains human-annotated real-speech predictions. The study systematically tested:

• Text-Based Detection: Using both oracle (reference-dependent) and reference-free metrics (e.g., perplexity, repetition rates) fed into tree-based ensemble classifiers.
• LLM-Based Detection: Utilizing models like GPT-4o-mini and Gemini 3.0, enhanced with domain-specific failure taxonomies and few-shot examples.
• Internal State Probing: Extracting and analyzing embedding sequences from the Whisper decoder layers using Bidirectional Long Short-Term Memory (BLSTM) classifiers.

Finally, the authors developed a late-fusion meta-classifier that combines the outputs of the best-performing text and internal-state models to maximize detection accuracy.

The study reveals that while text-based metrics are effective when ground-truth references are available, their performance collapses in zero-shot, reference-free settings. LLMs, despite their reasoning capabilities, incur high computational costs and fail to outperform simpler, lightweight classifiers. The most robust results were achieved by probing the internal decoder states of Whisper; specifically, middle-to-late decoder layers encode hallucination signals that can be effectively captured by a BLSTM. The final meta-classifier, which fuses these internal state signals with text-based features, achieved the highest F1 score (68.3%) and ROC AUC (90.0%), providing a scalable and accurate solution for real-world ASR deployment.