BEYOND FRAME SELECTION: RETHINKING LONG-VIDEO UNDERSTANDING WITH MLLMS

Long-video understanding is often limited by the fixed token budgets of Multimodal Large Language Models (MLLMs). Traditional approaches typically use uniform sampling or frame selection, which often force a trade-off between capturing broad temporal context and fine-grained visual details. The authors argue that this is a false dichotomy and that long-video understanding should instead be treated as a coordination problem between complementary evidence views.

To address this, the authors introduce VideoRouter (VR). First, they organize the video into a question-agnostic temporal hierarchy, partitioning the video into segments ranging from coarse (storyline-level) to fine (event-level). From this structure, they derive two distinct views: a global view, which uses representative center frames to capture broad event progression, and a local view, which uses query-guided pruning to isolate segments relevant to the specific question. Finally, a verification-guided router evaluates the evidence support for each view and selects the most reliable answer, or combines both views if neither is sufficient.

The authors demonstrate that a stronger standalone branch is not necessarily a better partner for the system, emphasizing that cross-view complementarity is more important than individual branch performance. Experiments on the VideoMME and LongVideoBench datasets show that VideoRouter consistently outperforms uniform sampling and state-of-the-art frame selection methods. Notably, on VideoMME, the method achieves a 2.9-point improvement over existing frame selection techniques using the LLaVA-Video-7B backbone, while also showing significant gains on LongVideoBench across multiple model architectures.

This work shifts the paradigm of long-video understanding from finding the single best set of frames to adaptively managing multiple evidence granularities. By explicitly modeling the trade-off between global context and local detail, the framework provides a more robust way to handle the diverse requirements of long-form video question answering. The ability to achieve superior performance with fewer frames compared to uniform sampling makes this an efficient and scalable solution for deploying MLLMs on long-duration video content.