Qi Xu, Yue Tan, Shihao Chen, Jiahao Meng, Anna Wang, Shunping Ji, Hao Fei, Jason Li
5 min
Abstract
Temporal Grounding (TG) aims to localize video segments corresponding to a textual query. Prior research predominantly focuses on single-segment retrieval. Real-world scenarios, however, often require localizing multiple disjoint segments for a single query -- a setting we term One-to-Many Temporal Grounding (OMTG). Previous state-of-the-art MLLMs, optimized for one-to-one settings, struggle in this context, often yielding near-zero scores due to a lack of event cardinality perception. To bridge this gap, we present a systematic solution with three key contributions. First, we establish the first comprehensive OMTG benchmark, introducing Count Accuracy (C-Acc) and Effective Temporal F1 (EtF1) as evaluation metrics. Second, we curate a high-quality OMTG dataset comprising 56k samples through a sophisticated construction pipeline. Third, we develop novel temporal and caption reward functions specifically designed for OMTG. In particular, the caption reward leverages Chain-of-Thought reasoning over dense video captions to explicitly guide policy optimization toward both preciseness and completeness. Extensive experiments show our model achieves a new state-of-the-art EtF1 of 43.65\% on OMTG Bench, outperforming Gemini 2.5 Pro and Seed-1.8 by 15.85\% and 15.61\%, respectively.
Alex: So it's a stricter grader. Getting the gist isn't enough; you have to show you understood the full structure of what happened.
Sam: That's the idea. And to train a model that can actually pass that stricter test, the authors use a specific learning method. Think of it like a coach watching several student attempts at the same problem and rewarding only the one that gets both the count and the timing exactly right. Over many rounds of practice, the model learns that guessing one big block is never acceptable—it has to find every distinct occurrence.
Alex: And I gather there's something extra that forces the model to explain itself, not just produce an answer?
Sam: Yes—they call it caption-guided reasoning. As the model identifies segments, it also has to describe what it's doing and why. If the written reasoning doesn't match the timestamps it selected, it loses points. It's the equivalent of a teacher asking a student to show their working. It prevents the model from arriving at a correct-looking answer through guesswork.
Alex: Is there a cost to all this extra scrutiny?
Sam: There is. The approach is computationally demanding. As videos get longer, the model has to hold more and more potential segments in mind simultaneously, and that becomes a real bottleneck. The authors are candid that scaling this to very long videos remains an open challenge.
Alex: So it's a trade-off—better counting accuracy, but at a meaningful cost in processing power.
Sam: That's a fair summary. Within the scope of their tests, though, the results are notable. Their model significantly outperformed larger proprietary systems that scored well on the old tIoU measure but failed badly once the stricter count-aware measure was applied.
Alex: Which suggests those larger models weren't actually solving the problem—they were just passing a test that wasn't strict enough.
Sam: That's the paper's broader point. The limitation wasn't purely about model size or raw computing power. It was about how the problem was defined in the first place. Once you reframe video search as "find all instances" rather than "find one instance," and once you measure success accordingly, a smaller, more carefully trained model can outperform much larger ones.
Alex: And the applications are fairly clear—sports analysis, medical imaging, security footage, anywhere you need to track a repeating event reliably.
Sam: Exactly. Whether it's counting every ball touch in a match, flagging repeated movements in a clinical scan, or logging every entry through a door, the framework points toward AI that genuinely understands a video's structure rather than just skimming it for a single highlight.
Alex: It's a useful reminder that sometimes the most meaningful progress comes not from building a bigger model, but from asking a more precise question. Thanks for listening to ResearchPod.