What Do Deepfake Benchmarks Measure? An Audit Using Frozen Self-Supervised Representations

As deepfake generators become increasingly sophisticated, the gap between performance on standardized benchmarks and real-world detection reliability has widened. This paper investigates whether current deepfake benchmarks are actually measuring forensic progress or if they are simply rewarding the general modality understanding already present in modern self-supervised learning (SSL) models.

The authors conduct an audit of image, audio, and video deepfake benchmarks using a deliberately simple diagnostic: they freeze the weights of state-of-the-art SSL models (such as V-JEPA2 for video, DINOv3 for images, and XLS-R for audio) and train basic linear probes (logistic regression and ridge classifiers) on the resulting representations. By keeping the backbone fixed, the researchers isolate the signal already present in generic representations from any task-specific forensic learning. They further analyze the geometry of these representation spaces using Fréchet distances to determine if the relative positioning of real and fake data distributions explains why some generators are harder to detect than others.

The audit reveals that linear probes on frozen SSL representations achieve performance levels competitive with, or in some cases superior to, specialized deepfake detectors across all three modalities. This indicates that a significant portion of the "progress" reported in current benchmarks is linearly accessible in generic feature spaces. Additionally, the authors find that generator-level difficulty is highly correlated with the Fréchet margin between the target generator and the source-fake distribution. This suggests that benchmarks may be testing how well a model maps new generators to already-seen spoof structures, rather than evaluating genuine forensic generalization.