Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model

Traditional automatic speech recognition (ASR) relies on autoregressive decoders that generate transcripts one token at a time, making inference time proportional to the length of the output. This paper investigates whether a large, frozen discrete-diffusion language model (DiffusionGemma) can be adapted to transcribe speech in parallel, potentially decoupling recognition speed from transcript length.

The researchers treat speech recognition as a diffusion process where the model refines a canvas of random tokens into a coherent transcript over a fixed number of denoising steps. To enable this, they keep the 26B-parameter DiffusionGemma backbone and a Whisper encoder frozen. They introduce a small, trainable projector to map Whisper acoustic features into the model's embedding space and use low-rank adapters (LoRA) to allow the frozen backbone to attend to the audio modality.

A critical challenge identified is the grounding deadlock: because the frozen language model has no prior knowledge of audio, the cross-attention mechanism initially ignores the audio features, preventing the projector from receiving useful gradients. The authors solve this by applying a connectionist temporal classification (CTC) loss directly through the output head, which forces the projector to align audio features with the transcript independently of the attention mechanism.

The resulting model achieves a 6.6% word error rate on LibriSpeech test-clean. By using a diffusion-based decoder, the model transcribes audio in approximately eight parallel steps, regardless of the length of the utterance. This approach demonstrates that a general-purpose diffusion language model can be effectively repurposed for speech recognition with only 42M trainable parameters (0.16% of the backbone), while maintaining multilingual capabilities through a single shared adapter.