Hallucinations Leave a Grounding Signature: Verifier-Guided Decoding for Selective Object Correction

Large vision-language models (LVLMs) often produce fluent, confident descriptions that include objects not present in the input image. Existing mitigation strategies—such as training-time alignment or post-hoc revision—often suffer from a trade-off: they reduce hallucinations but frequently shorten responses or suppress genuinely grounded objects. This paper asks whether it is possible to detect hallucinated objects during the generation process itself, allowing for selective correction that preserves the model's overall visual understanding.

The authors analyze the internal attention mechanisms of frozen LVLMs and discover the Intrinsic Grounding Signature (IGS). They observe that when a model generates an unsupported object, its attention heads exhibit a specific, structured redistribution: visual-grounding heads lose their alignment with image tokens (falling back to the sequence-start sink), while linguistic-continuation heads compensate by focusing more on the emerging text. By capturing these signed attention shifts, the authors train a lightweight, sparse linear verifier to assign a 'grounding risk' score to each emerging object mention.

Based on the IGS, the authors introduce Verifier-Guided Decoding (VGD). When the verifier identifies an object mention as high-risk, the system rolls back the KV cache to the start of that mention, suppresses the hallucinated term and its synonyms, and regenerates the continuation. Experiments on benchmarks like CHAIR and AMBER-G demonstrate that VGD achieves state-of-the-art hallucination reduction. For example, at a 90% recall threshold, VGD cuts AMBER-G CHAIR scores by 43.6% while retaining 99.6% of grounded-object coverage, all without shortening the overall caption length.

This work shifts the paradigm of hallucination mitigation from coarse-grained, model-wide interventions to precise, mention-level corrections. By proving that hallucinations leave a detectable 'signature' in the model's internal attention, the authors provide a practical, efficient way to improve the factual reliability of LVLMs without requiring expensive retraining or sacrificing the model's ability to provide detailed, descriptive outputs.