Data Safety: Synthetic Data Quality Analysis Using CIFAKE Dataset

As deep learning models increasingly rely on massive datasets, synthetic data has become a popular solution to address data scarcity and privacy concerns. However, synthetic images are not always equivalent to real data, and blindly incorporating them can lead to model performance degradation. This study investigates the quality of synthetic images by comparing two distinct generation methods (CIFAKE1 and CIFAKE2) against real images from the CIFAR-10 dataset.

The authors perform a multifaceted analysis of synthetic data quality across three dimensions: high-dimensional feature space (using CLIP, DINOv2, and DINOv3), low-level color statistics (illuminance and chrominance), and the model training process. By measuring the distance between synthetic and real data distributions, the researchers establish a correlation between data quality and classification performance. They also test practical mixing scenarios, such as replacing minority classes with synthetic data or blending synthetic and real data at varying ratios.

The study reveals that synthetic datasets with higher feature-space and statistical divergence from real data consistently underperform. Specifically, classes that show significant distance from real data centroids in UMAP/t-SNE projections exhibit higher misclassification rates. The authors identify that metrics like entropy and skewness in low-level color channels serve as reliable indicators of potential performance drops. By quantifying these gaps, the authors provide a strategy for practitioners to assess the quality of unknown synthetic data before training and determine the necessary amount of real data required to maintain model reliability.

This research provides a necessary safety framework for the integration of synthetic data in AI development. By moving beyond simple accuracy metrics to a data-centric evaluation of distribution alignment, it helps developers avoid the risks of "model poisoning" or performance degradation caused by low-quality synthetic inputs. This is particularly critical for high-stakes domains like healthcare or autonomous driving, where model reliability is paramount.