Chains That See, Answers That Don’t: A Multi-Aspect Evaluation Recipe for Forced Chain-of-Thought on Video-MME

This paper investigates whether forced chain-of-thought (CoT) prompting—where a vision-language model is instructed to generate reasoning steps before answering—actually improves performance on video question answering (VQA) tasks. The authors test the "boilerplate-chain" null hypothesis, which suggests that models might generate generic, input-invariant reasoning chains that influence answers through formatting effects rather than genuine visual understanding.

The authors introduce a three-probe evaluation recipe applied to the Qwen2.5-VL model family on the Video-MME benchmark. The probes include: (1) a comparison of accuracy across direct, CoT, answer-first, and no-video conditions; (2) a counterfactual video-swap diagnostic to measure how much reasoning chains change when the input video is replaced; and (3) a visual-degradation ladder to test how model performance scales as visual signal is systematically removed. The authors use both strict and permissive regex scorers to ensure that results are not artifacts of how the final answer is extracted.

The study finds that CoT chains are indeed sensitive to visual input; swapping the input video significantly alters the generated reasoning chains and the final answers, refuting the idea that these models rely on static, boilerplate chains. However, this visual conditioning does not translate into improved MCQ accuracy. In fact, for the 7B model, forced CoT leads to a statistically significant drop in accuracy compared to direct prompting. The authors conclude that while reasoning traces are grounded in the visual input, this does not guarantee better downstream performance, and the two should be evaluated as separate metrics.

This work challenges the assumption that forcing models to "think" before answering is a reliable way to boost performance in multimodal tasks. By demonstrating that grounded reasoning does not necessarily equate to higher accuracy, the authors provide a necessary framework for researchers to audit whether CoT is actually helping or merely adding unnecessary complexity to the inference process.