GeoThreat: Transferable Targeted Adversarial Attacks on Large Vision-Language Models for Remote Sensing Image Interpretation

As Large Vision-Language Models (LVLMs) are increasingly deployed for remote sensing tasks—such as disaster assessment and urban monitoring—their vulnerability to adversarial attacks has become a significant security concern. This paper addresses the challenge of performing targeted adversarial attacks on these models in black-box settings, where the attacker lacks access to the victim model's parameters. Specifically, the authors investigate how to manipulate remote sensing image interpretations to produce predefined, erroneous responses.

The authors propose GeoThreat, a method designed to overcome the limitations of existing attacks that often fail to account for the unique requirements of remote sensing, which demands joint reasoning over local discriminative cues and global scene context. GeoThreat employs a three-pronged strategy:

1. Conceptual Calibration: Using class tokens from surrogate image encoders to align the global semantic intent of the adversarial image with a target image.
2. Perceptual Adaptation: Utilizing a collaborative importance estimation approach to identify critical patch tokens and aligning them with target patch tokens via cross-attention, ensuring local visual cues are adapted toward the desired semantic details.
3. Ensemble-based Optimization: Iteratively updating adversarial perturbations by aggregating gradients across multiple surrogate models to enhance transferability across different LVLM architectures.

Extensive experiments demonstrate that GeoThreat outperforms state-of-the-art attack methods in both transferability and controllability. By effectively modulating both conceptual and perceptual representations, the method successfully steers the interpretations of diverse general-purpose and remote sensing-specific LVLMs toward designated target semantics. The study highlights that the joint optimization of global and local features is essential for bypassing the defenses of modern vision-language models in the remote sensing domain.