Breaking Shortcut Learning for Cross-Trial EEG-Guided Target Speech Extraction via Two-Stage Training

Research Problem

Recent advances in neuro-steered hearing technologies have enabled models to extract a target speaker's voice from a multi-talker mixture using EEG signals. While these models report high performance in standard within-trial evaluations, the authors identify a critical reliability bottleneck: these models often exploit trial-specific EEG patterns—such as slow drifts or electrode impedance trends—as shortcuts to identify the target speaker. Because these patterns are unique to specific recording sessions, the models fail to generalize when tested on unseen trials.

To address this, the authors introduce TRUST-TSE, a two-stage training framework designed to decouple representation learning from speech extraction.

• Stage 1 (Contrastive Pretraining): The EEG encoder is pretrained using a contrastive learning objective. To prevent the model from using trial identity as a shortcut, the authors implement 'attended-speaker negative sampling,' where negative samples are drawn from different segments of the same speaker. This forces the encoder to focus on fine-grained EEG-speech alignment rather than trial-specific cues.
• Stage 2 (Confidence-Weighted Extraction): The EEG encoder is frozen, and a speech extraction network is trained to condition on the learned embeddings. The authors introduce a confidence-weighted SI-SDR objective that uses the similarity between EEG and source signals to prioritize learning from segments where the neural guidance is most reliable.

Findings and Significance

Through rigorous cross-trial evaluation protocols on the KUL and DTU datasets, the authors show that standard end-to-end models perform near chance levels when evaluated on unseen trials. In contrast, TRUST-TSE maintains robust performance, demonstrating that the two-stage approach effectively breaks the reliance on trial-specific shortcuts. This work highlights the importance of rigorous cross-trial evaluation in neuro-steered technologies and provides a principled framework for developing more reliable, real-world-ready hearing aids.