T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Models

Research Question and Approach

Vision-language models (VLMs) like CLIP are highly vulnerable to adversarial perturbations, where small, imperceptible changes to an input image can cause significant classification errors. Existing defenses often require expensive retraining or indirect test-time adaptations, such as optimizing text prompts or pixel-space padding. The authors propose Test-time Visual Subspace Steering (T-VSS), a more direct and efficient defense that adapts the visual feature representation itself at inference time.

Instead of performing unconstrained updates in the full feature space, T-VSS constructs a sample-specific, low-rank subspace. By generating multiple augmented views of an attacked image, the authors observe that the feature residuals (the difference between augmented views and the original image) exhibit a shared, compact geometric structure. T-VSS uses Singular Value Decomposition (SVD) to isolate this low-rank subspace and learns a shared correction vector within it. This process is guided by reliability-weighted entropy minimization, which prioritizes stable, consistent views while suppressing outliers.

Experimental results across eight fine-grained datasets and various ImageNet benchmarks demonstrate that T-VSS consistently improves adversarial robustness compared to prior test-time adaptation methods. Across different CLIP backbones (ResNet-50, ViT-B/16, and ViT-L/14), T-VSS achieved the highest average adversarial accuracy while maintaining competitive clean accuracy. The authors highlight that because T-VSS optimizes only a small set of subspace coefficients rather than dense input variables or text prompts, it offers a superior robustness-efficiency trade-off.

This research provides a practical, lightweight alternative for securing VLMs in safety-critical applications. By shifting the focus from indirect prompt or pixel optimization to direct, geometry-aware feature steering, T-VSS demonstrates that adversarial robustness can be significantly improved without the need for additional training data or auxiliary models. This makes it a highly scalable solution for deploying robust vision-language systems in real-world environments.