ResearchPod Summary
Large Vision-Language Models (LVLMs) frequently suffer from object hallucination, where they generate non-factual content about input images. Existing mitigation strategies generally fall into two categories: training-based methods, which are computationally expensive and require extensive fine-tuning, and training-free methods. However, current training-free techniques either require multiple decoding rounds that add significant computational overhead or modify internal model states in a way that risks disrupting pretrained knowledge. This paper asks whether it is possible to achieve robust hallucination control during inference without altering model weights, performing multiple generation passes, or sacrificing computational efficiency.
The authors propose Test-Time Hallucination Mitigation (TTH), a training-free decoding framework that operates directly on token logits during generation. At each decoding step, the LVLM generates token probabilities, and a top-k candidate pool is evaluated. Using WordNet lexical resources, the framework isolates candidate tokens corresponding to physical objects. It then employs a zero-shot Multi-Modal Classifier—specifically CLIP—to compute visual grounding scores for these object candidates by matching object-specific prompts against the input image.
To bridge the scale difference between LVLM logits and CLIP similarity scores, TTH rescales the classifier outputs and combines them using an entropy-based weighting mechanism. The core intuition is that when the LVLM has high prediction entropy (indicating low confidence and a higher risk of hallucination), the system assigns greater weight to the visual grounding feedback from CLIP. Conversely, confident predictions rely more on the model's original logits. This selective adjustment is performed in a single forward generation pass, avoiding the heavy computational overhead of multi-round decoding strategies.
Extensive experiments across multiple benchmark datasets, including CHAIR, OPOPE, and LLaVA-Bench, demonstrate that TTH consistently improves accuracy and robustness across diverse LVLM families such as LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2. TTH outperforms existing baselines and weight-editing approaches in reducing object hallucinations while preserving caption quality and computational efficiency.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.