Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs

Multimodal Large Language Models (MLLMs) often suffer from visual laziness, a phenomenon where the model generates responses based on strong language priors rather than the actual visual content provided. This leads to hallucinations where the model's output contradicts the image. The authors investigate how to force these models to ground their reasoning in visual evidence rather than relying on linguistic shortcuts.

The authors introduce Visual Information Gain In aLignment (VIGIL), a post-training reinforcement learning framework. VIGIL uses a counterfactual approach: during training, it creates a 'blind state' by masking the attention between text and visual tokens. The model is then optimized to maximize the difference in likelihood between the 'seeing' state and the 'blind' state. By penalizing instances where the model remains confident even when it cannot 'see' the image, the framework forces the model to learn a causal dependency between visual features and the generated text.

VIGIL consistently outperforms standard alignment methods like Direct Preference Optimization (DPO) across various hallucination and reasoning benchmarks. Notably, it achieves these results using only 25% of the preference data required by other methods. Furthermore, the model demonstrates emergent spatial grounding capabilities—such as referring expression comprehension—without ever being explicitly trained on bounding box data. This suggests that teaching a model to avoid visual laziness inherently improves its spatial understanding.

Existing alignment methods often focus on the final text output, which can inadvertently reward models for being 'right for the wrong reasons'—using language patterns to guess the answer rather than analyzing the image. VIGIL provides a more fundamental, geometric constraint that ensures the model's reasoning is anchored in the pixel space. This approach is computationally efficient and provides a scalable way to build more reliable and truthful multimodal systems.