Do speech foundation models really learn words?

Self-supervised speech models (S3Ms) are widely used in speech recognition and speech-aware language models. While previous research has shown that these models can identify phonemes and words, it remains unclear whether their ability to identify words is merely a byproduct of their excellent encoding of phonological form (the sounds of words) or if they truly develop abstract, form-independent representations of word identity. This paper investigates whether these models encode word information that persists even when local phonological information is removed.

The authors employ a residualization technique to isolate the contribution of phonological information to model representations. By training a ridge regression to predict embedding features from phoneme, diphone, or triphone labels, they subtract these phonological components from the model's hidden representations. They then test whether a linear probe can still identify words from these 'residualized' embeddings. They also evaluate the impact of this process on downstream tasks like unsupervised word discovery (segmentation and clustering) to see if removing phonological noise helps the model focus on higher-level linguistic units.

The researchers find that even after removing phoneme and short-sequence phonological information, the later layers of HuBERT and wav2vec 2.0 retain a high capacity to identify words. This suggests that the models integrate context from surrounding frames to construct representations of words as units, rather than just encoding sequences of sounds. Furthermore, removing phonemic information through residualization improves performance in unsupervised word discovery tasks, indicating that the 'raw' representations contain phonological information that can interfere with higher-level lexical tasks.

This work provides a clearer understanding of what speech foundation models actually learn. By demonstrating that these models go beyond simple phoneme encoding to capture word-level information, the authors provide a theoretical basis for why S3Ms are effective in downstream linguistic tasks. The findings also suggest that targeted disentanglement of phonological information could be a useful strategy for improving the performance of speech models in low-resource or unsupervised settings.