ProCap: Prominence-guided Object Rectification for Faithful and Comprehensive Video Captioning

Video captioning models often struggle to balance linguistic fluency with visual accuracy, frequently hallucinating objects or omitting semantically significant details. While existing post-hoc rectification methods attempt to fix these errors by grounding captions in detected objects, they typically treat all objects as equally important and perform only a single, non-iterative correction pass. This paper asks: can we improve the faithfulness and completeness of video captions by prioritizing objects based on their visual and temporal prominence, and by refining captions through an iterative, multi-round process?

The authors introduce ProCap, a model-agnostic, post-hoc framework that enhances video captions without modifying the underlying vision-language model. The process begins by extracting key frames and generating initial captions, which are then rectified against detected objects. The core innovation is a prominence-aware scoring mechanism that ranks objects based on three factors:

1. Appearance: The relative size of the object within the frame.
2. Presence: How consistently the object appears across the video sequence.
3. Dynamics: The intensity of the object's movement and its relational interactions with other objects.

These scores are combined into a single prominence metric. The framework then uses an iterative refinement loop where an LLM is prompted to progressively incorporate high-prominence objects that were missing from the previous iteration of the caption. This continues until no further significant semantic gaps remain.

ProCap was evaluated on the MSVD and MSR-VTT datasets using object-grounded metrics and a 110-participant human study. The results demonstrate that the framework significantly outperforms strong pretrained baselines. Human evaluators reported up to a 48% increase in perceived completeness and a 45% reduction in hallucinations compared to standard captioning models. By prioritizing objects that are visually and narratively central to the video, ProCap produces more trustworthy and descriptive captions, making it a scalable solution for applications like accessibility and video retrieval.