What Counts as an Error? Dual-Reference Benchmarking for Atypical ASR

Automatic Speech Recognition (ASR) systems often struggle with atypical speech, such as that produced by people who stutter. A major, often overlooked challenge in evaluating these systems is the definition of ground truth. A single utterance can have two valid, yet distinct, transcriptions: a verbatim transcript that captures exactly what was said (including repetitions and prolongations) and an intended transcript that captures the speaker's underlying message with disfluencies removed. Most current research conflates these two, typically using intended transcripts as the default reference. This practice can be misleading, as a system optimized for one use case may perform poorly in another.

The authors benchmarked 11 ASR models from various architectural families—including encoder-decoder, CTC, and transducer models—using both verbatim and intended references. Their results demonstrate that model rankings are highly unstable when switching between these two references. For instance, a model that ranks first for verbatim accuracy might perform significantly worse for intended transcription.

This instability is not random; it reveals a clear architectural specialization. Autoregressive sequence-to-sequence models tend to excel at intended speech recognition, likely because their decoders use linguistic context to normalize acoustically uncertain regions. Conversely, CTC-based models, which rely more on local acoustic evidence, tend to perform better at verbatim transcription. This suggests that the choice of model architecture should be driven by the specific downstream application, such as clinical assessment (verbatim) versus voice-command interfaces (intended).

Choosing a single reference without disclosure effectively imposes a normative decision on how a person's speech should be represented. For clinicians and speech-language pathologists, verbatim transcripts are essential for analyzing stuttering patterns and severity. For users of voice assistants, intended transcripts are necessary for command execution. By failing to differentiate between these, the research community risks creating systems that "silently" normalize speech in ways that may be useful for some tasks but harmful for others. The authors advocate for explicit reporting of the intended use case and the use of both verbatim and intended metrics to ensure inclusive and accurate speech technology.