ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering

Existing multimodal large language models (MLLMs) often struggle with video text understanding because they rely on single-frame perception. Most current video-text datasets allow models to answer questions by looking at only one frame, failing to capture the temporal complexity of real-world scenarios where information is distributed across time. The authors address this by creating a dataset and training framework specifically designed to force multi-frame temporal reasoning.

The authors introduce ViTexQA, a dataset containing over 5,000 videos and nearly 7,000 QA pairs. Unlike previous benchmarks, ViTexQA is constructed through a rigorous, human-centric annotation pipeline that ensures every question is unanswerable from a single frame. The dataset includes temporally-grounded Chain-of-Thought (CoT) annotations that map textual evidence across timestamps. To improve model performance, the authors propose FrameThinker, a two-stage training approach: first, CoT-Guided Supervised Fine-Tuning (SFT) teaches the model to generate reasoning chains that explicitly link textual evidence across frames; second, Temporally-grounded Reinforcement Learning (RL) optimizes the model using rewards based on multi-frame coherence and accurate evidence aggregation.

Evaluations on ViTexQA reveal that state-of-the-art MLLMs perform significantly worse than humans, highlighting a fundamental gap in their ability to synthesize temporal information. The FrameThinker method effectively addresses this, outperforming strong baselines on the ViTexQA benchmark with a 6.3% improvement in ROUGE-L scores. These results demonstrate that explicit training for temporal perception is essential for moving beyond simple frame-level pattern matching in video understanding.

As video content becomes a primary source of information, the ability of AI to "read" and reason about text over time—such as tracking scoreboards in sports or following procedural steps in tutorials—is critical. This paper provides both the necessary evaluation infrastructure and a proven training methodology to advance the state of temporal perception in multimodal AI.