CORTIVA: Candidate-Score Fusion of Complementary Visual Teachers for EEG- and MEG-to-Image Retrieval

Decoding visual experiences from non-invasive brain activity like EEG and MEG is a core challenge in neuroscience. While recent methods use contrastive learning to align neural signals with visual representations, they typically consolidate heterogeneous visual information into a single embedding space before ranking. This paper investigates whether this early consolidation discards valuable, complementary information that could improve retrieval accuracy.

The authors propose CORTIVA, a framework that preserves the diversity of visual supervision by using three distinct 'decoding routes.' Each route is aligned to a different visual target (e.g., CLIP-RN50, composite visual representations, and ensemble projections). Instead of merging these into one embedding, CORTIVA allows each route to independently score a shared bank of candidate images. These independent score vectors are then combined using a learned router that weights the contributions of each route before the final ranking is determined.

CORTIVA significantly outperforms existing baselines on the 200-way THINGS-EEG2 benchmark, achieving 73.5% Top-1 accuracy—a 10.3 percentage point improvement over the strongest previous method. The framework also demonstrates strong performance on THINGS-MEG (42.4% Top-1). The authors show through matched route-removal experiments that the performance gain is specifically due to the integration of complementary score information rather than specialized weighting rules, confirming that preserving disagreement between encoders is beneficial for final retrieval.

This work shifts the paradigm of neural image retrieval from embedding-level consolidation to score-level fusion. By demonstrating that independent visual teachers provide complementary signals, CORTIVA offers a simple, modular, and effective way to boost decoding performance without requiring complex, monolithic architectures. This approach is highly interpretable and provides a clear path for integrating future visual models into brain-computer interface systems.