Entity-Faithful Repair of Synthetic Supervision for Zero-Shot Image Captioning

Zero-shot image captioning often relies on synthetic training data generated by text-to-image models. However, these synthetic pairs frequently suffer from 'structured fine-grained misalignment'—where the image and text appear globally plausible but fail to match at the level of specific entities or attributes. The authors investigate how to repair this synthetic supervision to improve the model's cross-modal grounding.

The authors introduce ReCap, a plug-and-play framework that shifts the focus from global image-text similarity to explicit entity-level realignment. The process consists of three main steps:

1. Entity-Guided Rewriting: The system uses an entity detector to identify which objects from the text are actually present in the synthetic image. It then uses a large language model (LLM) to rewrite the caption, incorporating retrieved textual cues to ensure the description is faithful to the visual content.
2. Adaptive Dynamic-Weighted Learning (ADWL): Recognizing that some synthetic pairs remain noisy even after rewriting, the authors introduce a weighting mechanism. This mechanism uses semantic consistency scores (calculated via Sentence-BERT) to downweight unreliable training samples during the captioning model's optimization.
3. Lightweight Training: The repaired, weighted dataset is used to train a standard image captioning model, which requires no additional retrieval or rewriting modules during inference.

ReCap consistently outperforms existing zero-shot methods across multiple benchmarks, including MSCOCO, Flickr30k, and NoCaps. By explicitly addressing entity-level discrepancies rather than relying on global similarity metrics, the model achieves state-of-the-art performance in both in-domain and cross-domain settings. The authors also demonstrate that ReCap is a versatile, plug-and-play framework that can be integrated into existing synthetic-data pipelines to improve their baseline performance.