Benchmarking the Alignment of Data-Quality Metrics, Human Judgment and Land-Cover Segmentation Performance for Earth Observation

Research Approach

This paper investigates the disconnect between common automated image quality metrics (such as FID, KID, and LPIPS) and the actual utility of synthetic data in Earth observation (EO) tasks. The authors argue that because these metrics are typically rooted in ImageNet-pretrained feature spaces, they are biased toward natural, object-centric imagery and fail to account for the unique top-down, orientation-agnostic nature of satellite data. To test this, the researchers conducted a three-way alignment analysis comparing automated metric scores, human perception (via a four-stage user study with 88 participants), and downstream semantic segmentation performance on mixed real-synthetic datasets.

Key Findings

The study reveals a stark misalignment between automated metrics and practical utility. First, semantics-preserving perturbations—such as rotation—drastically alter metric scores while leaving human recognition and semantic content unaffected. Second, the authors demonstrate that synthetic datasets with poor FID scores can still achieve high human-perceived realism and, crucially, improve downstream segmentation performance when used for data augmentation. Finally, the research shows that even real-world datasets from different geographic regions can receive worse automated metric scores than synthetic ones, suggesting that these metrics are fundamentally flawed for evaluating geospatial data quality.

As generative models become a standard tool for mitigating data scarcity in remote sensing, relying on standard fidelity metrics like FID can lead to misleading conclusions about data quality. This paper provides empirical evidence that these metrics are not reliable proxies for downstream task performance. The authors conclude that researchers should shift toward evaluation frameworks that prioritize downstream task utility and human-in-the-loop assessment rather than relying solely on distribution-based distance metrics.