The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning

Understanding Multimodal Reasoning

Vision-language models (VLMs) have become increasingly capable, yet they often struggle to maintain visual evidence as it moves through the language stack. This paper investigates this fragility by examining the internal dynamics of VLMs through a mechanistic lens. The researchers identify a stable, three-stage redistribution of multimodal attention focus across the depth of these models: an early phase of question-conditioned organization, a middle phase of visual-dominant relay, and a final phase of answer formation.

The Visual Relay Window (VRW)

The authors define the middle phase as the Visual Relay Window (VRW). By tracking attention probes—specifically how answer tokens look back to the question versus how visual tokens interact with one another—the study reveals that the geometry of this window is not fixed. Instead, it adapts to the specific demands of the task. For example, tasks requiring detailed perception exhibit broader, later-terminating relay windows, while reasoning-heavy tasks often require shorter, earlier-terminating windows. Mismatches between the task demand and the relay timing are shown to be a primary driver of hallucinations and ungrounded reasoning.

TRACE: Task-Adaptive Control

To address these failures, the authors propose TRACE (Task-adaptive Relay Anchoring and Controlled Evidence Scheduling). TRACE is an inference-time control framework that uses lightweight modules to predict and adjust the relay allocation during the prefill stage and preserve visual support during decoding. By explicitly controlling the relay geometry, TRACE allows the model to adapt its internal focus to the specific requirements of the input, leading to significant performance gains across multiple benchmarks, particularly in grounding-sensitive tasks.

This work provides a unified mechanistic explanation for why VLMs often fail to ground their answers in visual evidence. By shifting the focus from generic intervention techniques to a structural, task-aware control mechanism, the authors demonstrate that improving VLM reasoning does not necessarily require architectural changes, but rather a better management of the internal 'rhythm' of multimodal computation.