Local Margin Restoration for Test-Time Adaptation of Vision-Language Models

Vision-language models (VLMs) like CLIP often suffer from significant performance drops when faced with distribution shifts in real-world, unlabeled test streams. Existing test-time adaptation (TTA) methods often rely on entropy minimization, which can inadvertently reinforce incorrect predictions by aggressively sharpening the model's confidence. This paper investigates how to adapt VLMs effectively without access to source data or labels, specifically focusing on preventing the corruption of local semantic geometry and the accumulation of progressive bias that leads to model collapse.

The authors propose Local Margin Restoration (LMR), a one-step TTA framework designed around two core principles: preserving plausible near-top candidates and dynamically calibrating margins to prevent bias.

1. Protected Margin Restoration (PMR): Instead of forcing the model to commit to a single top-1 prediction, PMR identifies a set of plausible near-top candidates. It restores the semantic margin only against hard external competitors outside this set, effectively shielding the potential ground truth from suppression.
2. Dual-Stage Stabilization: To combat stream-level degradation, the authors introduce an Adaptive Margin (AM) controller that reduces the target margin for historically over-represented classes, and a Bias Correction (BC) module that calibrates residual output concentrations. This ensures that the model does not collapse into a few dominant classes over time.

Findings and Significance

Experiments on standard benchmarks (CIFAR-C, ImageNet-C) demonstrate that LMR consistently outperforms existing state-of-the-art TTA baselines. The framework is particularly effective in low-batch regimes where traditional methods struggle with noisy pseudo-labels. By explicitly modeling the local geometry of predictions and the temporal dynamics of the test stream, LMR provides a more robust and efficient way to maintain VLM performance in dynamic environments.