Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text

Spoken Language Models (SLMs) generate textual responses directly from speech by pairing a speech encoder with a large language model through a modality adapter. Despite strong downstream results, existing SLMs often exhibit weak instruction-following and limited task generalization compared to text-based language models. This occurs because current models map continuous, temporally varying speech into representations that remain structurally misaligned with discrete text embeddings. This paper investigates whether current SLMs process speech truly like text and proposes a framework to bridge this structural gap.

The authors analyze internal representations using Centered Kernel Alignment and token-wise similarity maps, revealing that standard SLMs maintain weak cross-modal alignment. To resolve this, the paper introduces a simple training framework with two core components. First, it uses a windowed Q-former adapter with a dynamic query allocation strategy during training. By allocating query tokens to match the exact target text length, the model decouples length mismatch from semantic alignment. Second, it incorporates a token-level internal alignment loss alongside standard behavior alignment, encouraging fine-grained correspondence between speech features and text representations across multiple layers of the LLM.

Evaluated across multiple benchmarks including AIR-Bench, SpeechR, MMSU, and Speech-IFEval, the proposed framework achieves competitive performance against strong baselines and closed-source models. Visualizations of token-wise similarity maps demonstrate clear diagonal patterns, confirming that explicit structural alignment creates a closer correspondence between speech and text. Ablation studies further validate that both dynamic query allocation and token-level internal alignment are essential for robust instruction-following and reasoning across diverse speech tasks.