Contrastive Mask Fidelity: Reference-Free Auditing of Ground-Truth Masks in Remote Sensing Semantic Segmentation

Semantic segmentation models are typically evaluated against human-drawn masks, assuming these references are perfect. However, in remote sensing, annotations are often coarse, misaligned, or incomplete, leading to an 'evaluation paradox' where models are penalized for producing masks that are more accurate than the ground truth. The authors introduce Contrastive Mask Fidelity (CMF), a training-free, reference-free metric designed to adjudicate between competing masks by evaluating them directly against the image evidence.

CMF operates by creating two counterfactual views of an image: a 'keep' view that isolates the region defined by a mask, and an 'erase' view that removes it. A frozen vision-language judge (RemoteCLIP) then scores these views to determine if class evidence is concentrated within the mask and absent outside it. This system, termed GeoVeritas, uses Seg-Probe—a strong, training-free candidate generator—to produce alternative masks for comparison against the human labels.

The authors audited 10,731 image-class pairs across ten remote-sensing benchmarks. The audit revealed systematic, class-dependent annotation distortion: man-made objects like buildings and roads frequently favored the model-generated candidate mask (62–85% of pairs), while ambiguous land-cover classes more often favored the human annotations.

CMF was validated against controlled mask corruptions and compared against expert human consensus. On a blinded three-annotator test, CMF matched expert judgment on 81% of pairs, outperforming simpler metrics like model confidence or keep-only scoring. Finally, the authors demonstrated that using CMF to arbitrate labels—selectively replacing human annotations with model masks—improved cross-domain transfer performance, confirming that the identified corrections provide higher-quality supervision than the original raw labels.

This work challenges the assumption that human-drawn ground truth is infallible. By providing a scalable, reference-free tool to audit and correct segmentation labels, the authors offer a path to improve model training in domains where high-quality, pixel-accurate manual annotation is prohibitively expensive or prone to systematic error. It shifts the focus of evaluation from mere agreement with a reference to actual faithfulness to the underlying visual evidence.