Brain-Aligned Multi-Stream Video Transformers with Sparse Self-Selection

Modern video transformers often rely on dense self-attention and monolithic architectures, which are computationally expensive and lack biological plausibility. This paper investigates whether incorporating neuro-inspired inductive biases—specifically pathway specialization and sparse competitive routing—can improve the efficiency, robustness, and biological alignment of video understanding models.

The authors propose a 'split-and-fuse' transformer architecture that mimics the primate visual system's ventral and dorsal pathways. The model consists of two parallel streams: a 'what' stream (high spatial resolution, low frame rate) for object recognition and a 'where' stream (low spatial resolution, high frame rate) for motion dynamics. To improve efficiency, the authors replace standard dense self-attention with a sparse 'winner-takes-all' token selection module. This module uses a Gumbel-softmax-based gate during training to select the most informative token per query, approximating the competitive inhibition observed in biological visual circuits.

The proposed model achieves competitive accuracy on Kinetics-400 and Something-Something V2 while operating on the Pareto frontier of accuracy versus inference time. The multi-stream design demonstrates increased robustness to spatial perturbations compared to standard single-stream baselines. Furthermore, representational similarity analysis (RSA) comparing model embeddings to human EEG recordings shows that the proposed architecture attains higher brain-model correlation than traditional video transformers, with the 'what' and 'where' streams showing patterns of activity consistent with their biological counterparts.

This work bridges the gap between high-performance computer vision and neuroscience. By demonstrating that neuro-inspired architectural choices—such as pathway specialization and sparse competition—can simultaneously improve engineering metrics (efficiency and robustness) and scientific metrics (brain-model alignment), the paper provides a blueprint for developing more interpretable and biologically grounded AI systems.