SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

Large Vision-Language Models (LVLMs) often suffer from hallucinations, where generated text is inconsistent with the provided visual input. While many existing methods attempt to fix this during the decoding stage, this paper argues that the root cause often lies in the visual tokens themselves—specifically, the presence of irrelevant or noisy tokens that mislead the model. The authors seek to address this by treating visual tokens as editable features rather than static inputs.

SeeMe introduces a three-stage "token engineering" process that operates during inference without requiring any model retraining or architectural changes:

1. Cross-Modal Selection: The model prunes semantically irrelevant visual tokens by analyzing cross-modal attention weights, removing noise early in the decoding pipeline.
2. Similarity-Guided Fusion: To prevent information loss from aggressive pruning, the model identifies locally similar tokens and fuses them to create a high-quality "semantic buffer" of enriched representations.
3. Final Refinement: The model performs a final selection step, retaining only the fused tokens that show the highest alignment with the textual context, ensuring that the final visual representation is both concise and faithful.

Experiments across four popular LVLMs (LLaVA-1.5, LLaVA-NEXT, INF-MLLM, and mPLUG-Owl2) demonstrate that SeeMe consistently reduces hallucinations across standard benchmarks like MME, POPE, and AMBER. The ablation studies confirm that while simple pruning can lead to information loss, the addition of the fusion and refinement stages effectively recovers fine-grained visual evidence. Furthermore, SeeMe maintains high computational efficiency, achieving the highest throughput among tested decoding strategies while introducing negligible memory overhead.