C3-Bench: A Context-Aware Change Captioning Benchmark

As the world is non-stationary, intelligent systems must be able to detect and describe changes between image pairs. However, existing benchmarks for change captioning are often limited to narrow, synthetic, or domain-specific datasets. This paper asks: how well do current models perform across diverse, real-world change contexts, and how can we build a more robust, context-aware evaluation framework?

The authors introduce C3-Bench, a comprehensive benchmark consisting of 4,996 human-labeled image pairs across 51 distinct real-world change contexts. These contexts are categorized into four domains: natural scenes, remote sensing, image editing, and anomalies. To move beyond rigid, reference-based metrics like BLEU or ROUGE, the authors implement an LLM-as-Judge framework that evaluates captions on correctness, specificity, fluency, and relevance. Additionally, they introduce a 'reversibility' metric to test whether models provide consistent descriptions when the order of the input image pair is swapped.

Key Findings

The study benchmarks 32 models, including conventional captioning models and various proprietary and open-source LMMs. The results demonstrate that conventional models are brittle and collapse when faced with contexts outside their training regimes. While LMMs show better generalization, they exhibit significant 'blind spots,' including systematic errors related to spatial reasoning and domain-specific nuances. Furthermore, the authors find that LMMs often struggle with input-order sensitivity, suggesting that their internal representation of change is not yet fully robust or symmetric.

By formalizing 'context-aware' change captioning, this work provides a necessary shift from generic image comparison to task-specific, grounded understanding. The findings highlight that simply scaling model size is insufficient for reliable change detection; instead, researchers must address the fundamental perceptual and logical failures that occur when models encounter novel, real-world scenarios. This benchmark serves as a critical tool for identifying these failure modes and guiding the development of more trustworthy AI systems.