Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings

Non-invasive brain-to-text decoding, such as using MEG or EEG, faces a persistent challenge: the representational mismatch between noisy neural signals and the semantic features used to decode them. Previous decoders typically rely on either static word embeddings (like Word2Vec) or dynamic contextual representations (like GPT) in isolation. This study investigates whether integrating these two feature types through a collaborative, non-linear fusion mechanism can better mirror human language processing and improve decoding performance.

The authors propose a multi-feature fusion framework that systematically benchmarks two integration strategies: linear Naive Concatenation and non-linear Multi-Head Cross-Attention. The framework uses a two-stage process: first, it reconstructs a unified semantic feature space from MEG recordings using self-supervised contrastive learning; second, it uses these reconstructed features to guide a beam-search text generation process. The cross-attention mechanism allows static lexical representations and dynamic contextual information to interactively modulate each other, simulating the cooperative pathways observed in human language comprehension.

The study demonstrates a clear performance hierarchy: Multi-Head Cross-Attention > Naive Concatenation > GPT-only > W2V-only. The non-linear cross-attention method achieved state-of-the-art results in semantic reconstruction and text generation. This suggests that neural language decoding is most effective when it captures the collaborative modulation between stable word meanings and evolving context, rather than treating these features as independent, parallel streams.

This research provides a robust, non-invasive method for brain-to-text decoding that overcomes the limitations of single-feature models. By aligning computational models more closely with the synergistic nature of human language networks, the framework offers a more reliable pathway for developing practical, non-invasive brain-computer interfaces (BCIs) that can reconstruct continuous natural language.