Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?

Building Speech LLMs typically involves connecting a pre-trained speech encoder to a frozen LLM via a trainable adaptor. A major challenge is the representational misalignment between the encoder, which is often trained on language-specific transcription tasks, and the LLM, which operates in a unified, language-agnostic semantic space. This paper investigates whether incorporating cross-lingual translation tasks into the speech encoder's pre-training can force the model to learn more universal, language-agnostic representations, thereby improving its integration with the LLM.

The authors compare three pre-training configurations for a speech encoder (based on the Whisper architecture):

1. ASR-Only: Multilingual transcription.
2. ASR & ST (X → en): Transcription plus translation from non-English languages to English.
3. ASR & ST (X ↔ en): Transcription plus bidirectional translation between English and other languages (Japanese, Chinese, and German).

To enable bidirectional translation, the authors introduced a redesigned decoder prompt format that explicitly conditions the task on the target language. After pre-training, the encoders were integrated into a Speech LLM pipeline with a frozen Llama-3.2 model, and performance was evaluated across ASR, speech translation (ST), intent classification, and emotion recognition tasks.

The results demonstrate that translation-enhanced pre-training consistently outperforms the ASR-only baseline. Specifically, the bidirectional (X ↔ en) configuration leads to superior cross-modal alignment. This improvement is not limited to the language pairs seen during pre-training; the model shows enhanced performance on unseen target languages, suggesting that the translation objective successfully encourages the encoder to decouple meaning from surface linguistic forms. These gains persist across both generative tasks and spoken language understanding tasks like intent classification.