Tengfei Liu, Yang Shi, Yuran Wang, Xiaohan Zhang, Yuqing Wen, Yuqi Tang, Qixun Wang, Zhuoran Zhang, Xuanyu Zhu, Weihong Lin, Xinlei Yu, Yujie Wei, Xinwei Long, Fengxiang Wang, Xinlong Chen, Yue Ding, Jialu Chen, Haotian Wang, Yuanxing Zhang
5 min
Abstract
Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captioning, a new task requiring factual video descriptions with phrase-level reference grounding, and propose RefCaptioner, a two-stage post-training framework for this task. RefCaptioner combines mixed-data SFT with Hierarchical Coverage-Discounted GRPO to jointly improve reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general video-captioning ability. To support training, we construct a corpus containing $20,000$ videos and 171,354 reference images. We further introduce MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding on both real-world and AI-generated videos. Experiments show that RefCaptioner achieves the best overall performance among the open-source models while remaining competitive on standard video captioning benchmarks. Human evaluation further confirms that its captions are preferred by annotators and enable more source-faithful video reconstruction with both open-source and proprietary video generators.
Sam: So the model gets rewarded when it cites correctly, and penalised when it doesn't?
Alex: Right. The specific method is called Hierarchical Coverage-Discounted GRPO — which is a technical name for a reward system with two distinct components. Think of it like a teacher grading an essay on two separate rubrics. One rubric — the Factuality Branch — asks: is the content of the caption accurate? The other — the Grounding Branch — asks: are the citations placed correctly, and do they point to the right images?
Sam: One part checks the story, the other checks the footnotes.
Alex: That's a good way to put it. And critically, if the model tries to cite a distractor — an image that isn't actually relevant to the video — the system penalises it specifically for that. So the model learns that it's better to acknowledge uncertainty than to invent a citation.
Sam: That's a meaningful constraint. It's building a kind of intellectual honesty into the system.
Alex: That's a fair description. There's also a third component that handles the grouping problem — checking whether the model correctly associates all the citations for the same person, rather than scattering them inconsistently across the caption. The researchers call this Cross-Reference Semantic Coherence.
Sam: Does the approach actually hold up when tested?
Alex: The paper reports that RefCaptioner achieves the strongest performance among open-source models on this task. Importantly, it also maintains its quality on standard captioning benchmarks — meaning it doesn't sacrifice its ability to describe a video clearly in order to gain the ability to cite its sources. That balance is one of the study's central claims.
Sam: And what are the limits? What happens if the reference images themselves are poor quality, or incomplete?
Alex: That's a significant constraint the paper acknowledges. The model relies entirely on the quality of the images it's given. It cannot invent visual details that aren't present in the reference set. So the system is only as reliable as the evidence it receives — which is an honest limitation, but a real one. The researchers suggest that future work might focus on systems that can autonomously retrieve their own reference images, rather than depending on a pre-supplied set.
Sam: So right now it needs someone to hand it the right photos. But the longer-term vision is a system that goes and finds its own evidence.
Alex: That's the direction the paper points toward — captioning that functions more like an evidence-retrieval process than a description exercise. For now, RefCaptioner is a meaningful step toward AI systems that don't just describe what they see, but can be held accountable for how they know it. Thanks for listening to ResearchPod.