Interleaved Speech Language Models Latently Work In Text

This paper investigates the internal mechanisms of interleaved speech-text language models (SLMs) to understand how they process speech and text modalities. Using the 'logit lens' technique—which projects intermediate hidden states into the model's output vocabulary—the authors analyze the latent representations of speech inputs across various transformer layers. They specifically evaluate whether these models, which are trained on a mix of speech units and text tokens, develop a shared latent space that facilitates cross-modal reasoning.

The researchers discovered that interleaved SLMs exhibit a consistent three-stage processing pattern. In the early layers, the model processes speech tokens. In the middle layers, the representations shift into a text-like latent space where the model implicitly transcribes the spoken words into text tokens. Finally, in the later layers, the model shifts back to the speech domain to generate output. Remarkably, the correct textual transcription of the spoken word is decodable from these intermediate layers for up to 77% of the data, even though the models were never trained with a supervised speech-to-text objective.

Beyond simple transcription, the authors found that the model's 'textual workspace' also supports predictive reasoning. By analyzing the logit-lens outputs, they demonstrated that the model can predict the next word in a sequence—and even retrieve factual answers—while still processing the speech input. This suggests that the model is effectively 'thinking' in text to bridge the gap between speech input and semantic knowledge retrieval. The study concludes that both pre-training on text and the use of interleaved training data are essential for this behavior to emerge.

Understanding that SLMs operate through an implicit textual transcription phase provides a new perspective on how these models achieve semantic capabilities. It suggests that the performance gains observed in interleaved models are not just a result of increased data, but a fundamental shift in how the model represents and processes information. This insight could lead to more efficient training strategies and better architectural designs for future speech-processing systems.