HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems

Research Overview

Modern Automatic Speech Recognition (ASR) systems often exhibit hallucinations—erroneous text generations that do not correspond to the input audio. While these errors can lead to significant misinformation, existing research has primarily evaluated mitigation strategies using non-speech audio or artificially corrupted data. This paper introduces HALAS (Hallucination Annotations for Large-scale ASR Systems), the first human-annotated benchmark for naturally occurring hallucinations in seven state-of-the-art ASR models using real earnings call recordings.

The researchers curated the HALAS dataset by selecting audio segments from the Earnings 22 corpus that triggered high inter-model disagreement, as these segments are most prone to ASR errors. Ten professional annotators performed span-level labeling to identify hallucinations, loopings, and phonetic errors, achieving high inter-annotator agreement. The authors then used this dataset to evaluate various detection strategies, including proxy metrics (like Word Error Rate and Perplexity), LLM-based reference checking, and decoder-embedding (DE) classification.

Key Findings

The analysis confirms that all seven tested ASR models produce hallucinations on real speech, with error distributions heavily skewed toward a small set of frequent phrases. Notably, hallucinations frequently occur even when the transcription has a low Word Error Rate, making them difficult to detect with standard structural metrics. While proxy metrics achieved an ROC-AUC of up to 81%, state-of-the-art detection methods, including LLM-based approaches, achieved an F1 score of only 53.1%. The study demonstrates that multi-layer decoder-embedding detectors trained on HALAS provide the most robust performance, suggesting that the dataset is a valuable resource for training generalizable hallucination detectors.

By providing a rigorous, non-artificial benchmark, this work shifts the focus of ASR safety research from synthetic noise to real-world deployment challenges. The findings highlight that current ASR systems are not as reliable as their low error rates might suggest, and that specialized detection methods are necessary to ensure the integrity of automated transcriptions in critical domains.