It’s the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling

Test-time scaling—sampling multiple candidate solutions and selecting the best one—is a standard technique for improving LLM reasoning. However, this approach often fails for Vision-Language Models (VLMs) because the selection layer cannot distinguish between an image-grounded answer and a confident guess derived from the language prior. This paper investigates whether Perturbation-Grounded Selection (Pgs), which scores candidates based on their stability across label-preserving image perturbations (e.g., cropping, masking, jitter), can provide a reliable, training-free signal for selecting the correct answer.

The authors identify a critical confounding factor in previous evaluations: comparing Pgs against chain-of-thought (CoT) majority voting conflates the perturbation signal with a change in decoding format (CoT vs. short-form answers). To isolate the effect of the perturbation, they introduce a format-matched control (MatchedCtrl). This control uses the same number of short-form samples as Pgs but draws them from the original, unperturbed image. By comparing Pgs to MatchedCtrl, the authors isolate whether the perturbation-based reweighting actually provides visual grounding or if the performance gains are merely due to the inclusion of additional short-form samples.

The study reports a negative result: while Pgs shows significant gains over standard CoT-only majority voting (up to +31.8 points on TextVQA), these gains vanish when compared to the format-matched control. Across all tested benchmarks, including vision-intensive tasks like ViLP, Pgs performs no better than MatchedCtrl. Furthermore, while the authors observe a real, image-dependent stability gap, this gap does not predict whether Pgs will select the correct answer on a per-instance basis. The authors conclude that perturbation consistency is a diagnostic of visual dependence but not a usable selection signal for test-time scaling.