Steering Vision-Language Models with Joint Sparse Autoencoders

Research Approach

This paper introduces the Joint Sparse Autoencoder (JSAE) to address the difficulty of steering vision-language models (VLMs) using standard sparse autoencoders. While traditional sparse autoencoders (SAEs) decompose model activations into interpretable features, they often fail to provide cross-modal control because vision and language representations remain misaligned. The authors propose a framework that uses an explicit cosine-similarity constraint during training to force the encoders of both modalities to map inputs into a shared, aligned latent space. This allows the researchers to extract text-side latent directions and inject them into the visual stream to steer model generation.

Key Findings

The researchers applied JSAE to three prominent VLMs: LLaVA-v1.6-Mistral-7B, Llama3-LLaVA-8B, and Qwen3-VL-30B. They discovered a distinct layer-dependent asymmetry in how these models respond to interventions. Additive steering—where a specific concept vector is injected to influence output—is most effective at mid-to-late layers (specifically around layer 25 in LLaVA-7B) and weakens significantly as the model approaches the final output layers. In contrast, feature suppression (inhibiting a concept) remains relatively consistent across all probed layers. The authors demonstrate that these aligned sparse representations provide significantly more control over multimodal features than unconstrained alternatives.

Understanding the internal mechanisms of VLMs is critical for improving their reliability and controllability. By identifying specific layers where visual and linguistic information are most effectively aligned, this work provides a roadmap for researchers to perform targeted interventions. The JSAE framework offers a practical method for steering model behavior without the need for expensive fine-tuning, potentially enabling safer and more predictable multimodal AI systems.