Ill-Posed by Design: Probing Evidence Use in VLMs

This paper investigates how vision-language models (VLMs) perform on monocular metric object-size estimation—a task where physical dimensions cannot be determined from a single image without calibration. The authors ask whether VLMs truly use visual evidence (such as scene geometry, local context, or apparent size) or if they rely on internal language-based category priors to guess object sizes.

The researchers introduce Metric VQA, a benchmark consisting of over 10,000 queries from the Objectron dataset and 331 tape-measured, in-the-wild scenes. They evaluate 12 open-weight VLMs (ranging from 3B to 397B parameters) using a suite of counterfactual visual interventions. These interventions systematically mask or perturb specific evidence channels—including target identity, local context, apparent size, and global scene geometry—to observe how each change affects the model's output. They also compare these results against a text-only frontier LLM that has no access to visual input.

The study reveals that even the most powerful open-weight VLMs struggle to leverage visual cues effectively. On in-the-wild scenes, no open-weight VLM outperforms a text-only frontier LLM, suggesting that models are largely ignoring the visual evidence they are provided. The diagnostic signatures show that target identity is the most load-bearing cue, while global scene geometry is almost entirely ignored. Furthermore, while fine-tuning with LoRA improves performance, it does not teach the models to use geometric reasoning; instead, it simply reinforces the specific cues (like category priors) that the base model already favored.

This work demonstrates that high performance on VQA benchmarks can be misleading. By framing metric estimation as an ill-posed diagnostic, the authors expose that current VLMs are not truly performing spatial reasoning but are instead relying on strong linguistic biases. This highlights a critical gap in the development of multimodal systems: the inability to integrate visual geometry into quantitative reasoning, even at massive scales.