Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision

Visual state tracking remains a significant challenge for Video Large Language Models (Video-LLMs), which often struggle to maintain accurate running counts and scene states as objects are added, removed, or occluded. Existing training objectives typically prioritize action recognition or moment localization, failing to incentivize the cumulative reasoning required for persistent state tracking. This paper asks whether a model can learn to track visual states over time using only unlabeled video data and self-generated supervision.

The authors introduce S3T (Self-Supervised Self-Distillation over Time), a framework that treats temporal sampling density as privileged information. The core hypothesis is that a denser sampling of a video clip provides a more accurate representation of the running scene state than a sparse sampling. S3T uses a dense-view teacher (24 frames) to distill its next-token distribution into a sparse-view student (12 frames) using the same frozen base model and a single trainable LoRA adapter. The model generates its own targets, eliminating the need for ground-truth labels, external judges, or reward models. The training objective combines this distillation term with a reference anchor to keep the adapted model calibrated to the base model.

S3T significantly improves cumulative-state reasoning on the VSTAT benchmark, increasing overall accuracy by +2.70 points. These gains are concentrated on axes requiring evidence accumulation, such as Count, Atomic, and Sequence tracking. The capability learned from unlabeled synthetic clips transfers to real-world videos, yielding improvements of +7.95 on VSTAT-YouTube cumulative-state questions and +4.50 on MVBench Action Count. Crucially, these improvements are achieved without sacrificing general video-understanding performance, and the method adds no additional inference cost.