TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition

Adapting large-scale Vision-Language Models (VLMs) like CLIP for video recognition often leads to a performance trade-off: the model gains task-specific knowledge but loses the broad, open-vocabulary generalization capabilities learned during pretraining. This paper investigates why standard fine-tuning fails to maintain this balance, specifically focusing on how optimization within a limited training distribution causes representation drift and alignment shifts in out-of-distribution (OOD) space.

The authors identify that standard fine-tuning objectives are restricted to the training distribution, which leads to suboptimal alignment when the model encounters unseen categories. To address this, they introduce TACO, a framework that enforces consistency across the entire representation space.

TACO employs two primary mechanisms:

1. Relative Structure Distillation (RSD): Instead of relying on expensive OOD data, the authors use randomly sampled geometric anchors on the CLIP hypersphere. By distilling the relative relationships between these anchors and the model's visual/text embeddings, TACO forces the model to maintain a consistent geometric structure even for concepts outside the training set.
2. Specialization Projection: To prevent the model from over-fitting its core representations to the training data, the authors decouple the representation space from the optimization space. A lightweight, residual linear projection is used during training to absorb task-specific bias, which is then discarded during inference to preserve the original, more generalizable representation space.

TACO demonstrates that preserving the relative geometry of the embedding space is critical for OOD generalization. By regularizing the entire space rather than just the training distribution, the model avoids the "over-shifting" of cross-modal alignments that typically degrades performance on novel categories. The authors show that their approach achieves state-of-the-art results across multiple cross-dataset and base-to-novel video recognition benchmarks, confirming that their simple geometric anchors are sufficient to provide a stable reference for adaptation.