MissingBench-Verified: Probing Vision-Language Models’ Inability to Detect Missing Object Parts

This paper investigates a specific, high-stakes failure mode in state-of-the-art (SOTA) Vision-Language Models (VLMs): the inability to recognize when an essential component of an object has been removed from an image. While prior research has documented general object hallucinations, this study focuses on whether models can override their internal 'real-world' expectations (e.g., that an iPhone must have a home button) when presented with visual evidence to the contrary.

The authors introduced MissingBench-Verified, a benchmark consisting of 118 human-verified images where essential parts were removed using advanced image-editing techniques. They evaluated ten leading proprietary and open-source VLMs using a variety of strategies: direct querying, pre-answer and post-answer injection of 'perfect' external detection tool results, and allowing models to use image-processing tools (like cropping and contrast adjustment) to inspect the scene. They also compared these results against traditional object detection models like OWLv2 and YOLOE.

The study reveals that all ten tested models are highly susceptible to hallucinations, with none achieving an accuracy higher than 75% in identifying missing parts. The models exhibit a strong 'existence bias,' where they prioritize their internal parametric knowledge over both the visual input and explicit external evidence. Surprisingly, providing models with access to image-processing tools or increasing their reasoning effort (Chain-of-Thought) provided negligible improvements. Unlike VLMs, traditional object detection models did not hallucinate the missing parts but instead struggled with semantic confusion or failed to detect objects entirely, suggesting that the VLM failure is a deeper, model-specific issue of over-reliance on internal priors.

These findings highlight a fundamental limitation in current VLM architectures for inspection and monitoring tasks. Because these models are 'stubborn'—refusing to update their beliefs even when provided with contradictory evidence—they are currently unreliable for safety-critical applications where verifying the absence of a component is as important as verifying its presence. The authors argue that simple prompting or post-hoc corrections are insufficient, and that future progress will require architectural or training-level interventions to help models override internal biases.