Reliability-Guided Adaptive Ensembling for Robust Test-Time Adaptation

Test-time adaptation (TTA) allows models to adapt to new domains without source data, but it is highly vulnerable to adversarial contamination. Malicious inputs in the test stream can corrupt the model's online updates, leading to catastrophic performance degradation. This paper investigates how to make TTA methods robust to such adversarial streams without requiring source data or model retraining.

The authors propose SAFER (Stochastic Augmentation Framework for Enhanced Robustness), a modular wrapper that can be applied to existing TTA methods. Instead of relying on a single, potentially corrupted input, SAFER generates multiple stochastic augmentations for each test sample. It then estimates the reliability of each view based on feature-space agreement with other views. The framework uses a 'cc_drop' mechanism to identify and discard the least reliable view (often the one containing the strongest adversarial signal) and pools the remaining predictions using reliability-weighted aggregation. An optional extension, SAFER-A, allows for adaptive mixing between the original view and the augmented ensemble to better preserve performance on clean data.

SAFER consistently improves the robustness of various TTA baselines across multiple benchmarks (PACS, VLCS, and OfficeHome) under black-box PGD adversarial attacks. By replacing brittle single-view predictions with reliability-guided ensembles, the framework prevents corrupted samples from destabilizing the online adaptation process. The authors demonstrate that the robustness gains stem specifically from the reliability-guided aggregation and outlier suppression, rather than simply from the use of multiple augmentations.