Is EEG-to-Text Feasible in Real-World Scenarios? An In-Depth Analysis Using a Neuropsychology-Inspired Benchmark

Recent progress in EEG-to-text (EEG2Text) decoding has been clouded by skepticism regarding whether these models actually learn from neural signals or simply rely on the linguistic priors of large language models. The authors investigate why current systems fail to generate meaningful text without 'teacher-forcing'—a training technique that feeds ground-truth tokens to the model during inference—and whether this reliance is an artifact of poor benchmark design rather than an inherent limitation of EEG data.

The researchers identify 'EEG instability'—the stochastic, non-stationary nature of neural signals across different sessions and cognitive states—as the primary confounder in existing benchmarks like ZuCo. To address this, they developed the Corpus of EEG-to-Text (COFETT), a high-density (128-channel) dataset featuring repeated trials of identical sentences. This design allows the model to isolate invariant linguistic components from trial-to-trial noise. Additionally, they replace traditional token-level metrics (like BLEU) with a vector-alignment approach, which evaluates the semantic similarity between EEG embeddings and text embeddings without requiring teacher-forcing.

Comparative experiments reveal that on existing benchmarks, models perform identically whether fed real EEG data or random noise, confirming that these benchmarks fail to measure genuine neural decoding. In contrast, the COFETT benchmark successfully discriminates between models based on their ability to extract linguistic information from EEG. The authors provide statistical evidence that EEG signals contain recoverable linguistic content, provided that the evaluation protocol accounts for signal instability and avoids the 'shortcut' of teacher-forcing.

This work shifts the focus of the EEG2Text field from architecture-heavy approaches to the fundamental reliability of evaluation protocols. By proving that EEG contains decodable linguistic information, the study provides a roadmap for developing non-invasive brain-computer interfaces that could eventually offer a practical, non-surgical communication alternative for patients with severe paralysis.