Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do

This paper investigates the effectiveness of multimodal Chain-of-Thought (CoT) reasoning, a technique designed to improve Large Multimodal Models (LMMs) by eliciting step-by-step thinking. The authors seek to determine what these models can and cannot do, where they fail, and why. To do this, they evaluate 14 non-reasoning models and 8 reasoning models across 12 distinct tasks, categorized into perception (e.g., OCR, visual grounding, object counting) and reasoning (e.g., mathematical, scientific, logical, spatial) domains.

The researchers identify three primary insights. First, CoT is not universally beneficial; it is a selective tool. While it improves performance in mathematical and scientific reasoning, it often introduces distracting, unnecessary steps in perception-oriented tasks, leading to performance drops in areas like visual grounding and object counting.

Second, the authors find that current open-source multimodal reasoning models provide only marginal improvements over their base models. They suggest this is likely due to an over-reliance on training data focused on mathematical problems, which neglects broader reasoning capabilities.

Third, the study highlights a "Look Light, Think Heavy" pattern. During the reasoning process, models show a decline in visual reflection while verbal reflection remains active. This indicates that current models struggle to maintain deep visual introspection throughout extended reasoning, making visual reasoning a significant bottleneck for future progress.

As LMMs are increasingly deployed for complex, real-world tasks, understanding the limitations of CoT is critical. This paper demonstrates that simply increasing the length of a model's reasoning chain does not guarantee better performance if the model lacks the ability to maintain focus on visual input. The findings suggest that future development should focus on improving visual introspection and integrating external tools to bridge the gap between verbal reasoning and visual understanding.