ROSE: BENCHMARKING THE PERCEPTION-TO-ACTION GAP IN MULTIMODAL MODELS

As multimodal large language models (MLLMs) are increasingly tasked with interacting with visual environments, a critical question arises: can these models reliably convert visual interpretations into context-specific actions? While models have shown proficiency in visual recognition and description, it remains unclear whether they can maintain this performance when the same visual evidence requires different actions based on changing task constraints.

The authors introduce ROSE (Reference-conditioned Oddity and Symbolic Execution), a benchmark designed to isolate the perception-to-action interface. Unlike existing benchmarks that evaluate models on fixed questions or puzzles, ROSE holds the visual scene constant while varying the task context. Each scene consists of a grid of visually similar elements, requiring the model to first infer a majority reference and identify sparse exceptions. The benchmark then queries the model through five coupled tasks, ranging from global counting (e.g., "count all exceptions") to region-conditioned coordinate actions (e.g., "click only the exceptions within this specific region").

Evaluating nine recent MLLMs, the researchers found a consistent and substantial performance drop when moving from counting to action-oriented tasks. Even when models correctly identified the number of exceptions in a global context, they often failed to perform the correct coordinate-level actions when restricted by specific regions. This gap persists even when models are capable of correct cardinality, suggesting that the bottleneck is not merely visual recognition, but the ability to translate that recognition into precise, context-dependent symbolic execution. Human performance on the same tasks remains near-perfect (98.8%), highlighting that the failure is specific to current model architectures.