Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

Research Question and Approach

Video Object Segmentation (VOS) typically requires dense, manual annotations, which are expensive and difficult to scale. This paper introduces Cross-Temporal Consistency and Clustering (CTC2), a self-supervised framework designed to learn video object representations without human labels. Instead of focusing on whole objects or individual pixels, the authors propose a mid-level approach that treats objects as collections of semantically meaningful parts (e.g., wheels, limbs, or articulated components).

The framework utilizes a frozen SAM2 vision transformer backbone to extract spatial tokens. To manage computational complexity and focus on relevant information, the authors implement an adaptive token selection strategy that uses the transformer's [CLS] attention maps to identify salient, spatially diverse regions. These selected tokens are then passed through a lightweight MLP-based clustering head to generate soft part assignments. Temporal consistency is enforced by matching these part assignments across multiple time offsets using a symmetric Kullback–Leibler (KL) divergence loss, which encourages stable part-level alignment without requiring optical flow or synthetic motion cues.

Key Findings

CTC2 achieves competitive performance on standard benchmarks like DAVIS-2016, DAVIS-2017, and YouTube-VOS. By avoiding dense warping and external memory modules, the framework maintains real-time throughput. The authors demonstrate that their part-level clustering approach is robust to occlusion and viewpoint changes, as these mid-level abstractions persist more reliably than pixel-level features. Furthermore, the model shows strong generalization capabilities across different datasets and can be easily adapted to a semi-supervised setting by using a first-frame mask.

This research offers a scalable, label-free alternative to supervised VOS methods. By leveraging the emergent properties of large pretrained transformers (like SAM2) and focusing on part-level consistency, the authors provide a pathway for video understanding that is both computationally efficient and robust to the limitations of traditional motion-based or pixel-centric approaches. The design is particularly relevant for applications where manual annotation is impractical or where objects undergo significant deformation and occlusion.