Average Rankings Mask Per-Subject Optimality: A Friedman–Nemenyi Benchmark of EEG Motor-Imagery BCI Decoders

Researchers often claim that a specific decoding pipeline—typically a spatial or Riemannian method—is the superior choice for motor-imagery brain-computer interfaces (BCIs). This study investigates whether such a universal best pipeline actually exists. By testing over 1,000 distinct decoding configurations under the most favorable conditions possible—where models are trained and tested within the same session for each individual—the authors determine if a single method consistently outperforms others or if individual variability necessitates a more personalized approach.

The authors utilized the Mother of All BCI Benchmarks (MOABB) framework to evaluate 1,056 unique pipelines (combinations of feature extractors, scalers, and classifiers) across three public motor-imagery datasets. The analysis involved over 340,000 subject-level model fits. To ensure statistical rigor, the study employed Friedman omnibus tests and Nemenyi critical-difference analysis to compare the performance of six representative pipelines. The researchers specifically examined whether the ranking of these pipelines was consistent across different cohorts or if it was driven by feature dimensionality.

While spatial and Riemannian methods (specifically covariance tangent-space projection and Common Spatial Patterns) generally outperformed other families, their relative ranking was highly dataset-dependent. On the largest and most heterogeneous dataset, these two leading methods were statistically indistinguishable. Furthermore, the study revealed that the "best" pipeline is highly participant-specific; the top-performing method for the group was only the best choice for about one-third of individuals. Matching the pipeline to the specific participant could improve accuracy by approximately seven percentage points, suggesting that the field's focus should shift from finding a universal decoder to developing participant-aware model selection strategies.