Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captioning, a new task requiring factual video descriptions with phrase-level reference grounding, and propose RefCaptioner, a two-stage post-training framework for this task. RefCaptioner combines mixed-data SFT with Hierarchical Coverage-Discounted GRPO to jointly improve reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general video-captioning ability. To support training, we construct a corpus containing $20,000$ videos and 171,354 reference images. We further introduce MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding on both real-world and AI-generated videos. Experiments show that RefCaptioner achieves the best overall performance among the open-source models while remaining competitive on standard video captioning benchmarks. Human evaluation further confirms that its captions are preferred by annotators and enable more source-faithful video reconstruction with both open-source and proprietary video generators.
Alex: Welcome to another episode of ResearchPod. Today, we're discussing a research paper that introduces a framework called RefCaptioner — a system designed to make AI models prove they're actually looking at the right visual evidence when they describe a video.
Sam: So it's about making sure the AI isn't just guessing? Like, actually forcing it to show its work?
Alex: Exactly. Think about how a good student cites sources in an essay. Current AI video-description models can write fluent, detailed captions — but there's no requirement for them to point to the specific images that back up what they're saying. They might describe a person accurately, or they might just be making a plausible guess. There's no way to tell.
Sam: Right. If I hand an AI a video and a set of reference photos, how do I know it's actually using those photos as evidence, rather than just producing something that sounds convincing?
Alex: That's the core problem. So the researchers created a new task — they call it "multi-reference image-grounded video captioning." The idea is straightforward: whenever the model mentions a specific object or person in its description, it must immediately insert a tag — essentially a digital footnote — that points to the reference image it used.
Sam: So every claim in the caption has to be backed up by a citation. Like a research paper, but for video descriptions.
Alex: Precisely. And that citation requirement is what separates genuine understanding from what researchers call "hallucination" — where an AI confidently describes something that isn't actually supported by the visual evidence in front of it.
Sam: I imagine this gets complicated fast. What if the model is given images that look relevant but actually aren't in the video?
Alex: That's one of the central difficulties. The researchers deliberately include what they call "distractors" — reference images that look plausible but don't actually correspond to anything in the video. The model has to learn to filter those out. And on top of that, it has to correctly group multiple photos of the same person together, even if those photos look quite different from one another.
Sam: So it's not just about finding the right image — it's about not being fooled by the wrong ones, and recognising that two different photos might show the same person.
Alex: Exactly. Both of those are genuinely hard problems. And the researchers use a two-stage approach to teach the model to handle them.
Sam: What does that look like in practice?
Alex: The first stage is fairly conventional — they train the model on a carefully curated set of human-verified examples, so it learns the basic structure of the task. The second stage is where it gets more interesting. They use a technique from reinforcement learning — the kind of training where a system learns by receiving rewards for good behaviour and penalties for bad behaviour, the way you might train a dog, or the way a video game character learns to navigate a level.
Sam: So the model gets rewarded when it cites correctly, and penalised when it doesn't?
Alex: Right. The specific method is called Hierarchical Coverage-Discounted GRPO — which is a technical name for a reward system with two distinct components. Think of it like a teacher grading an essay on two separate rubrics. One rubric — the Factuality Branch — asks: is the content of the caption accurate? The other — the Grounding Branch — asks: are the citations placed correctly, and do they point to the right images?
Sam: One part checks the story, the other checks the footnotes.
Alex: That's a good way to put it. And critically, if the model tries to cite a distractor — an image that isn't actually relevant to the video — the system penalises it specifically for that. So the model learns that it's better to acknowledge uncertainty than to invent a citation.
Sam: That's a meaningful constraint. It's building a kind of intellectual honesty into the system.
Alex: That's a fair description. There's also a third component that handles the grouping problem — checking whether the model correctly associates all the citations for the same person, rather than scattering them inconsistently across the caption. The researchers call this Cross-Reference Semantic Coherence.
Sam: Does the approach actually hold up when tested?
Alex: The paper reports that RefCaptioner achieves the strongest performance among open-source models on this task. Importantly, it also maintains its quality on standard captioning benchmarks — meaning it doesn't sacrifice its ability to describe a video clearly in order to gain the ability to cite its sources. That balance is one of the study's central claims.
Sam: And what are the limits? What happens if the reference images themselves are poor quality, or incomplete?
Alex: That's a significant constraint the paper acknowledges. The model relies entirely on the quality of the images it's given. It cannot invent visual details that aren't present in the reference set. So the system is only as reliable as the evidence it receives — which is an honest limitation, but a real one. The researchers suggest that future work might focus on systems that can autonomously retrieve their own reference images, rather than depending on a pre-supplied set.
Sam: So right now it needs someone to hand it the right photos. But the longer-term vision is a system that goes and finds its own evidence.
Alex: That's the direction the paper points toward — captioning that functions more like an evidence-retrieval process than a description exercise. For now, RefCaptioner is a meaningful step toward AI systems that don't just describe what they see, but can be held accountable for how they know it. Thanks for listening to ResearchPod.