Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigation

Vision-language models (VLMs) often generate fluent descriptions that include objects not present in the input image—a phenomenon known as object hallucination. This is largely driven by the model's reliance on learned language-vision priors rather than visual evidence. The authors propose "QK Product Steering," a training-free, data-free, and zero-inference-cost method to mitigate this. Instead of modifying decoding or fine-tuning the model, the authors perform a one-shot weight edit. They identify the dominant singular modes of the query-key (QK) product—the operator that determines attention logits—in middle layers and suppress them. This edited product is then mapped back to the query weights using a closed-form update that preserves shared key weights, ensuring compatibility with Grouped-Query Attention (GQA).

Across three different GQA-based VLMs (Qwen2.5-VL-7B, InternVL3-8B, and Pixtral-12B), QK Product Steering achieved an average relative reduction in CHAIRs (a metric for object hallucination) of 4.0%. The authors found that suppressing dominant modes in the middle layers is most effective, as these layers exhibit a structural profile where the edit is "surgical"—meaning it removes hallucination-prone signals while leaving the majority of the attention operator's energy intact. Furthermore, by decomposing the QK product into symmetric (mutual content-similarity) and antisymmetric (directional attention) components, the study revealed that the hallucination signal is primarily localized to the symmetric channel. Random-mode controls showed negligible impact, confirming that the hallucination signal is specifically tied to these dominant spectral modes.

This approach provides a highly practical alternative to existing hallucination mitigation strategies. Because it is a one-shot weight edit, it avoids the computational costs associated with training-based methods and the latency penalties of inference-time interventions like contrastive decoding or logit adjustment. By demonstrating that hallucination can be mitigated through a targeted, low-rank spectral edit, the paper offers a mechanistic understanding of how vision-language priors manifest in attention layers and how they can be surgically dampened to improve model reliability.