Discriminative Axis, Not Data Volume: What a Contrastive Corpus Teaches an Audio Embedding

When a contrastive model fails to learn a specific attribute, the standard industry response is to scale the training data. This paper challenges that assumption by documenting a case where adding large amounts of lexical speech data significantly improved keyword spotting but actively degraded speech-emotion recognition. The authors demonstrate that this is not a capacity limit—the model is physically capable of representing both—nor is it a lack of data volume. Even when the training corpus is expanded with thousands of clips where captions explicitly name the target emotion, the model fails to learn the attribute.

The core insight is that contrastive objectives are "lazy": they only encode features that are strictly necessary to distinguish an anchor from its in-batch negatives. If a model can separate two audio clips based on scene content or lexical keywords, it has no incentive to learn the more subtle prosodic cues required for emotion recognition. The authors show that in standard mined datasets, captions are diverse enough that scene-level differences provide an easy shortcut. Consequently, the model ignores the emotion-related information in the audio, even if that information is explicitly mentioned in the text.

To prove this, the authors perform a controlled intervention. They show that by using a prosody-controlled corpus—where lexical content is held constant across different emotions—the model is forced to rely on prosody to minimize the contrastive loss. This structural change, rather than the volume of data, is the primary driver of attribute encoding. By collapsing caption diversity and forcing emotion to be the only discriminative axis, the researchers successfully recovered the model's ability to recognize emotion without changing the underlying audio data or the model's capacity.