Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering

As Vision-Language Models (VLMs) become standard components in modern AI pipelines, they are increasingly distributed as pre-trained artifacts (weights, architecture definitions, and computation graphs) via third-party hubs. This paper investigates whether a malicious provider can exploit this supply-chain trust by embedding dormant, trigger-activated "architectural backdoors" that remain invisible during standard performance evaluations but alter model behavior upon command.

The authors propose a method called representation steering. Instead of poisoning training data or manipulating model weights, they modify the model's architecture definition to include a small, trigger-gated additive payload. This payload consists of two parts: a trigger detector that monitors input tokens and a precomputed steering vector. When the specific trigger is absent, the steering vector is zero, and the model functions normally. When the trigger is present, the vector is added to the model's intermediate representation, shifting the internal state toward an attacker-defined objective (e.g., forcing a specific output or bypassing safety filters).

The researchers successfully implemented this attack across several prominent VLM families, including CLIP, BLIP, LLaVA, and Qwen-VL. The backdoor was shown to be effective across diverse tasks, including visual question answering, text-to-image generation, and semantic response biasing. Crucially, the attack preserves the model's utility on clean, non-triggered inputs, allowing it to pass standard validation benchmarks. The authors also demonstrate that existing auditing practices—which typically focus on inspecting learned weights—are insufficient to detect this type of malicious executable logic, and they propose a new defense mechanism that inspects the model's computational graph and hidden-state trajectories.