AdvNav: Behavior-Guided Black-Box Adversarial Attacks on Vision-Language Navigation

As Vision-and-Language Navigation (VLN) systems are increasingly deployed in real-world environments, their safety and robustness against adversarial attacks have become critical. Most existing adversarial attacks against these agents require white-box access to model gradients, which is often impossible in proprietary or deployed systems. This paper asks: can we effectively disrupt a VLN agent's navigation using only observable inputs and outputs, without access to internal model gradients?

The authors propose AdvNav, a black-box adversarial attack framework that injects spatially coherent, Perlin-noise-based perturbations into the agent's visual input stream. Because the attacker cannot compute gradients, AdvNav uses a dual-granularity feedback mechanism to guide the optimization of these perturbations. This feedback includes:

• Trajectory-level performance: A score based on navigation success rate, path efficiency, and navigation error.
• Action-level reward: A score measuring the drift in action probabilities at each step, capturing latent decision errors.
• Deviation indicator: A binary signal flagging when the agent deviates from the intended path.

This feedback guides a hybrid optimization strategy that combines adaptive tuning of perturbation strength with genetic evolution of the noise structure, allowing the attacker to iteratively discover the most disruptive noise patterns.

AdvNav was evaluated against two prominent VLN architectures: the Transformer-based HAMT and the LLM-based MapGPT. The framework demonstrated significant effectiveness, achieving an Attack Success Rate (ASR) of 49.70% against HAMT, and 65.96% and 87.30% against MapGPT using different backbones (Qwen3-VL and GPT-4V, respectively). These results indicate that current VLN models are highly susceptible to structured visual perturbations, even when the attacker has no knowledge of the model's internal parameters or gradients.