A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP

As deep learning models become increasingly vulnerable to adversarial attacks, existing detection methods often fail because they require access to the classifier's architecture, knowledge of the specific attack mechanism, or large amounts of labeled adversarial data. The authors ask: can we build a robust, zero-shot adversarial detector that is completely agnostic to both the classifier and the attack type?

The authors propose A4D (Attack- and Architecture-Agnostic Adversarial Detector), which leverages the pre-trained CLIP model. The core insight is that while adversarial perturbations are often imperceptible to humans, they induce a measurable, non-arbitrary shift in CLIP's shared image-text embedding space.

To detect these shifts, the framework:

1. Defines a dictionary of text prompts describing various forms of noise and corruption.
2. Computes the cosine similarity between an input image and these prompts.
3. Uses a small, unlabeled set of clean images (a "representative set") to estimate the distribution of these similarity scores.
4. Applies Principal Component Analysis (PCA) to aggregate these similarities into a single detection score, identifying images that deviate from the expected clean distribution.

A4D achieves state-of-the-art detection performance across multiple datasets (Tiny-ImageNet, StreetSurfaceVis) and diverse classifier architectures (DeiT-Small, Wide-ResNet, ResNet34, ConvNext-Tiny). Because it operates in the embedding space of a foundation model rather than relying on classifier-specific features, it remains effective against a wide range of attacks, including FGSM, PGD, AutoAttack, and Square, without needing to be retrained for each scenario.

This work demonstrates that foundation models like CLIP contain rich, non-semantic information that can be repurposed for security tasks. By moving away from attack-specific or architecture-specific defenses, A4D provides a practical, efficient, and highly generalizable tool for securing vision systems against emerging threats in real-world, black-box environments.