Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shift

The Challenge of AI Detection

AI text detection is typically treated as a supervised learning problem, where models are trained on labeled datasets of human-written and AI-generated text. However, this approach is inherently fragile because it assumes the data distribution at test time will match the training data. In practice, AI text detection faces constant, post-deployment distribution shifts: adversarial humanization (where users rewrite AI text to evade detection), the rapid release of new LLM models, and natural temporal shifts in human writing styles. Because supervised models are fixed after training, they are easily outmaneuvered by these evolving targets.

A New Paradigm: Test-Time Adaptation

This paper proposes shifting from static supervised learning to test-time adaptation (TTA). The core insight is that while individual AI-generated texts may be difficult to classify, LLM usage in the wild often exhibits "inference-time homogeneity"—a consistent statistical signal across a batch of samples. By using semi-supervised learning techniques, specifically Positive-Unlabeled (PU) and Positive-Negative-Unlabeled (PNU) learning, the detector can adapt its decision boundary using unlabeled data collected at inference time. This allows the model to "move last" and adjust to the evader's latest strategies without requiring new labeled data.

Empirical Performance

The authors demonstrate that while state-of-the-art supervised detectors (including commercial models like Pangram) perform well on in-distribution data, their accuracy collapses under shift. For example, in adversarial settings, Pangram's detection rate dropped to 24.1%, whereas the proposed TTA approach maintained 90.5% accuracy. The study shows that TTA methods are robust across multiple scenarios, including natural shifts between different LLM versions and temporal drifts in human writing, proving that leveraging latent homogeneity is a powerful strategy for maintaining detection performance in the wild.