Rethinking Object-Centric Representations for Video Dynamics Modeling

Unsupervised video object tracking aims to decompose dynamic scenes into persistent, object-centric entities without manual annotations. Existing slot-based models often struggle with this task because they entangle appearance and pose within a single latent representation. When these models enforce temporal consistency to track objects, the objective conflicts with natural object motion and viewpoint changes, leading to fragmented masks, identity swaps, and a tendency for slots to lock onto static background regions.

The authors propose STAITUS (Sparse and Temporally Aligned InvarianT Unsupervised Slots), a framework that explicitly disentangles each slot into two components: an appearance embedding and a geometric pose (position and scale). This disentanglement allows the model to enforce temporal alignment only in the appearance space, while letting geometric attributes evolve freely. Additionally, the framework incorporates an adaptive gating mechanism that dynamically adjusts the number of active slots based on scene complexity, preventing the over-segmentation common in models with a fixed slot count.

STAITUS demonstrates superior performance in both segmentation quality and tracking stability across synthetic and real-world benchmarks. By decoupling appearance from pose, the model produces sharper object masks and maintains more persistent identities during complex scenarios, such as object occlusion, entry, and exit. Ablation studies confirm that the combination of geometric disentanglement, adaptive slot selection, and spatio-temporal regularization is essential for robust, unsupervised object-centric representation learning.

This work addresses a fundamental limitation in unsupervised video understanding. By enabling models to distinguish between what an object looks like and where it is located, STAITUS provides a more reliable foundation for downstream tasks like video prediction, causal reasoning, and robotic planning, where maintaining object identity through motion is critical.