How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation

Automated fact-checking (AFC) systems typically follow a two-stage pipeline: retrieving relevant evidence and then predicting the veracity of a claim based on that evidence. While many models achieve high scores on individual benchmarks, it remains unclear whether these systems possess generalizable capabilities or if they simply exploit dataset-specific artifacts. This paper provides a comprehensive cross-benchmark evaluation of nine models—ranging from simple sparse baselines to state-of-the-art systems—across four structurally diverse datasets spanning scientific, open-web, and climate domains.

The study reveals three critical insights. First, system rankings are highly dependent on the domain and evaluation metric; a model that performs exceptionally well on one dataset (e.g., SciFact) may drop to near-baseline performance on another (e.g., ClimateCheck). Second, classical baselines like logistic regression over sparse retrieval (TF-IDF) remain surprisingly competitive, often outperforming complex zero-shot LLMs and top-ranked shared-task systems when evidence retrieval is noisy. Third, retrieval quality is the primary bottleneck for veracity prediction. When researchers replaced retrieved evidence with gold-annotated evidence, accuracy improved by 14–22 percentage points across all models, demonstrating that current veracity models are limited more by the quality of the retrieved information than by their own reasoning capabilities.

These findings suggest that current leaderboard rankings in AFC research may be misleading indicators of real-world utility. The authors argue that the field must move toward cross-domain evaluation, mandate the use of simple classical baselines as lower bounds, and explicitly decouple the evaluation of retrieval and veracity components. Without these rigorous standards, it is impossible to distinguish genuine progress in language understanding from dataset-specific engineering or the exploitation of annotation noise.