Temporal Grounding (TG) aims to localize video segments corresponding to a textual query. Prior research predominantly focuses on single-segment retrieval. Real-world scenarios, however, often require localizing multiple disjoint segments for a single query -- a setting we term One-to-Many Temporal Grounding (OMTG). Previous state-of-the-art MLLMs, optimized for one-to-one settings, struggle in this context, often yielding near-zero scores due to a lack of event cardinality perception. To bridge this gap, we present a systematic solution with three key contributions. First, we establish the first comprehensive OMTG benchmark, introducing Count Accuracy (C-Acc) and Effective Temporal F1 (EtF1) as evaluation metrics. Second, we curate a high-quality OMTG dataset comprising 56k samples through a sophisticated construction pipeline. Third, we develop novel temporal and caption reward functions specifically designed for OMTG. In particular, the caption reward leverages Chain-of-Thought reasoning over dense video captions to explicitly guide policy optimization toward both preciseness and completeness. Extensive experiments show our model achieves a new state-of-the-art EtF1 of 43.65\% on OMTG Bench, outperforming Gemini 2.5 Pro and Seed-1.8 by 15.85\% and 15.61\%, respectively.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a specific limitation in how artificial intelligence watches video.
Sam: We're discussing a paper titled "Towards One-to-Many Temporal Grounding." The central puzzle is that while current AI models are reasonably good at finding a single event in a video, they struggle when the same event happens multiple times.
Alex: So the question is: why do AI models fail when they have to find every instance of a repeating action?
Sam: Exactly. Imagine you have a two-hour security feed and you ask the AI to find every time a door opens. A standard model might find the first opening and stop. Or it might merge all the separate openings into one long, incorrect block of time. The paper calls this failure "cardinality blindness"—the model simply cannot count how many times something happens.
Alex: That's like asking someone to count how many times a player touches the ball in a soccer match, but they only record the first touch and ignore everything after.
Sam: That's a good way to put it. The paper argues that current models treat video search as a simple retrieval task—like finding one specific needle in a haystack—rather than what it actually needs to be: a set-generation task, where you identify every single needle present.
Alex: If the models are failing at this, how are researchers measuring the failure? I assume the standard way of checking accuracy doesn't capture it.
Sam: You're right, and this is where things get interesting. Traditionally, researchers use a measure called "tIoU"—Temporal Intersection over Union. It checks how much a predicted time segment overlaps with the actual event. The more overlap, the higher the score. But as this study shows, a model can score very well on that measure while still being completely wrong.
Alex: How can you score well and still be wrong?
Sam: Because tIoU only cares about total overlap, not whether the events were counted separately. If an event happens four distinct times, but the model draws one giant block covering all four, the overlap might look fine on paper. The model got the rough timing right but completely missed the fact that there were four separate events. It's like a student who writes down the right total time but doesn't notice the class had four separate breaks instead of one long one.
Alex: So the measurement tool itself was hiding the problem.
Sam: Precisely. To fix that, the authors introduce a new measure called the "Effective Temporal F1-score," or EtF1. It requires the model to get both the timing and the count correct. If the count is wrong, the score drops to zero—no partial credit for getting the timing close.
Alex: So it's a stricter grader. Getting the gist isn't enough; you have to show you understood the full structure of what happened.
Sam: That's the idea. And to train a model that can actually pass that stricter test, the authors use a specific learning method. Think of it like a coach watching several student attempts at the same problem and rewarding only the one that gets both the count and the timing exactly right. Over many rounds of practice, the model learns that guessing one big block is never acceptable—it has to find every distinct occurrence.
Alex: And I gather there's something extra that forces the model to explain itself, not just produce an answer?
Sam: Yes—they call it caption-guided reasoning. As the model identifies segments, it also has to describe what it's doing and why. If the written reasoning doesn't match the timestamps it selected, it loses points. It's the equivalent of a teacher asking a student to show their working. It prevents the model from arriving at a correct-looking answer through guesswork.
Alex: Is there a cost to all this extra scrutiny?
Sam: There is. The approach is computationally demanding. As videos get longer, the model has to hold more and more potential segments in mind simultaneously, and that becomes a real bottleneck. The authors are candid that scaling this to very long videos remains an open challenge.
Alex: So it's a trade-off—better counting accuracy, but at a meaningful cost in processing power.
Sam: That's a fair summary. Within the scope of their tests, though, the results are notable. Their model significantly outperformed larger proprietary systems that scored well on the old tIoU measure but failed badly once the stricter count-aware measure was applied.
Alex: Which suggests those larger models weren't actually solving the problem—they were just passing a test that wasn't strict enough.
Sam: That's the paper's broader point. The limitation wasn't purely about model size or raw computing power. It was about how the problem was defined in the first place. Once you reframe video search as "find all instances" rather than "find one instance," and once you measure success accordingly, a smaller, more carefully trained model can outperform much larger ones.
Alex: And the applications are fairly clear—sports analysis, medical imaging, security footage, anywhere you need to track a repeating event reliably.
Sam: Exactly. Whether it's counting every ball touch in a match, flagging repeated movements in a clinical scan, or logging every entry through a door, the framework points toward AI that genuinely understands a video's structure rather than just skimming it for a single highlight.
Alex: It's a useful reminder that sometimes the most meaningful progress comes not from building a bigger model, but from asking a more precise question. Thanks for listening to ResearchPod.