Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation

Continual Test-Time Adaptation (CTTA) aims to update models on unlabeled, shifting data streams after deployment. However, because source data is often unavailable due to privacy or licensing, models frequently suffer from catastrophic forgetting and compounding errors as they adapt to long-term distribution shifts. This paper asks: can we create a compact, privacy-preserving summary of the source data to act as a stable anchor for long-term adaptation?

The authors introduce DO-ALL (Distill Once, Adapt Life-Long), a plug-and-play framework that uses Dataset Distillation (DD) to summarize the source domain into a tiny set of synthetic anchors. Before deployment, the source dataset is distilled into these anchors, which store synthetic inputs, their soft labels, and latent feature representations. During deployment, the model matches each incoming target sample to its most semantically similar anchor. This correspondence enables three stabilization strategies: anchor-based replay (to preserve source knowledge), manifold-smoothing regularization (to ensure stable decision boundaries), and harm-adaptive blending (to selectively revert unstable parameter updates toward the source initialization).

DO-ALL consistently improves the robustness of existing CTTA algorithms across major benchmarks, including CIFAR100-C, ImageNet-C, and the large-scale CCC benchmark. The authors demonstrate that the distilled anchors provide a persistent, memory-efficient reference that effectively counters the drift typically seen in source-free adaptation. Furthermore, they observe a direct positive correlation between the semantic quality of the distilled anchors and the resulting stability of the model during adaptation, confirming that these synthetic points act as a reliable "anchor" for the model's representation.