ResearchPod Summary
Large Vision-Language Models (LVLMs) frequently generate text that is linguistically plausible but inconsistent with the provided image, a phenomenon known as hallucination. While previous research has attributed this to the dominance of language priors over visual inputs, the specific mechanistic origin of these priors within the transformer architecture has remained unclear. This study investigates where these priors originate and how they interfere with visual evidence during the decoding process.
The authors decompose transformer computations into attention and Feed-Forward Network (FFN) contributions using a residual stream perspective. By applying logit lens projections to LLaVA-1.5, they track how predictions evolve across layers. They find that attention modules consistently aggregate visual evidence, whereas FFN modules in specific critical layers (typically 16–22 in 7B models) act as the primary source of language priors. These priors can override the visual evidence established in earlier layers, causing the model's output to drift toward incorrect, hallucinated tokens. Based on this, they introduce FADE (FFN Attenuation for DEcoding), a training-free intervention that scales down FFN outputs at these critical layers to suppress language-prior dominance.
FADE effectively mitigates hallucinations across a wide range of benchmarks, including POPE, CHAIR, and MME, while preserving the model's general capabilities. Unlike contrastive decoding methods that require multiple forward passes or complex perturbations, FADE operates in a single forward pass with minimal latency overhead. Experiments across diverse architectures—including LLaVA-1.5, mPLUG-Owl2, InstructBLIP, InternVL3, and Qwen-VL—demonstrate that FADE provides a more stable and efficient hallucination-mitigation trade-off than existing training-free baselines.
This work provides a mechanistic explanation for why LVLMs hallucinate, shifting the focus from output-level interventions to targeted internal component manipulation. By demonstrating that FFN modules are the primary source of harmful language priors, the study offers a lightweight, architecture-agnostic strategy for improving the reliability of multimodal models in high-stakes applications like medical diagnosis and autonomous driving, where precision is critical.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.