Automating Learner Assessment: Benchmarking Machine Learning and Deep Learning Models for EEG-Based Familiarity Prediction

Objective assessment of learning remains a major challenge in education, as traditional examinations are often poor proxies for true cognitive understanding. This paper investigates whether electroencephalography (EEG) can provide a direct, non-invasive measure of knowledge acquisition by predicting cognitive familiarity. The authors explore how machine learning models perform when tasked with classifying whether a learner is familiar with specific stimuli—specifically human faces (representing factual knowledge) and mathematical equations (representing conceptual knowledge).

The researchers recorded continuous EEG data from 23 healthy participants using a 14-channel setup while they viewed familiar and unfamiliar faces and mathematical equations. From the preprocessed EEG epochs, 84 spectral features were extracted across six frequency bands using Power Spectral Density. A central methodological contribution of the paper is exposing the illusion of high accuracy caused by temporal data leakage. When models are evaluated using standard Stratified K-Fold cross-validation—where overlapping sliding-window epochs from the same trial end up in both training and testing sets—performance appears near-perfect. To counter this, the authors implement a rigorous Group K-Fold protocol where epochs are grouped by unique trial blocks, ensuring models are tested on entirely unseen trials.

Fifteen machine learning and deep learning models were benchmarked. Under standard stratified cross-validation, models achieved stellar results, with the Convolutional Neural Network reaching an F1-score of 0.9853. However, under the rigorous trial-independent Group K-Fold protocol, the peak performance dropped to 0.6038 (using the CNN), and Gradient Boosting reached 0.5510. Despite this substantial drop, the independent evaluation scores remained statistically significant above the 25% chance level. Secondary analyses using SHAP and feature importance further revealed that temporal and frontal Gamma and Beta oscillations are the most critical biomarkers driving familiarity predictions.