Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens

As self-supervised learning (SSL) models move toward using discrete tokens from neural audio codecs (NACs) to reduce computational costs, a critical question arises: does the language used to train the NAC or the SSL model itself dictate cross-lingual performance? This paper systematically disentangles these two stages to determine if NACs need to be retrained when moving between languages.

The authors conducted a controlled study using Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER) as downstream tasks across English, Japanese, and Chinese. They decoupled the process into three research questions:

1. How does the NAC training language affect reconstruction quality?
2. How does the SSL pre-training language affect downstream performance when the NAC is fixed?
3. How does the NAC training language affect performance when the SSL pre-training language is fixed to the target language?

They utilized the Descript Audio Codec (DAC) and HuBERT-based SSL models, comparing performance across various language-specific training conditions.

The experiments revealed that the NAC training language has a negligible impact on downstream task performance. Even when a NAC was trained on a different language, it maintained stable performance upon reconstruction. Conversely, the SSL pre-training language is a primary driver of performance; aligning the SSL pre-training data with the target language is essential for optimal results. These findings suggest that practitioners can reuse a single, pre-trained NAC across multiple languages, provided they perform language-specific SSL pre-training.

This research provides a practical roadmap for scaling speech SSL models. By confirming that NACs do not require language-specific retraining, the authors demonstrate that developers can significantly reduce the storage and computational overhead of building multilingual speech systems. It clarifies that the "language sensitivity" of these models is primarily a property of the SSL pre-training phase rather than the initial audio discretization.