VanillaBench: The Hidden Accuracy Cost of Adversarial Robustness

Research Question and Approach

Adversarial robustness research has long focused on the trade-off between a model's performance on clean, unperturbed data and its resilience against adversarial attacks. However, most literature reports these metrics in isolation, failing to quantify the 'accuracy cost'—the drop in standard performance compared to a non-adversarially trained (vanilla) model. This paper introduces VanillaBench, a systematic benchmark designed to make this hidden cost explicit by comparing 186 robust models from the RobustBench leaderboard against multiple vanilla baselines.

The author evaluates these models across four threat models (CIFAR-10, CIFAR-100, and ImageNet) using several reference points: the best and median standard models from Papers with Code, a temporal best (the best vanilla model available at the time of the robust model's publication), and an architecture-matched baseline to isolate the impact of training methods from architectural differences.

The results reveal that the accuracy cost of robustness is universally negative and often substantial. Across all tracks, no robust model matches the performance of the best vanilla models. Even when controlling for architecture, robust models trail their vanilla counterparts by a mean of 3.5 to 17.5 percentage points. The study further demonstrates that this is not merely a limitation of weaker models; even the top-10 most robust models per track exhibit significant performance deficits, ranging from 4.6 to 21.2 percentage points below vanilla references.

This research highlights a critical transparency gap in the current robustness literature. For practitioners and decision-makers, the accuracy cost of adversarial training is a primary factor in deployment feasibility. A small drop in accuracy might be an acceptable 'insurance premium' for security, but a double-digit drop can render a model commercially unviable. By establishing a standardized way to report these gaps, VanillaBench provides the necessary context for stakeholders to make informed decisions about whether the robustness gains justify the performance sacrifices.