QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment

Video instance segmentation (VIS) typically relies on video-level supervision to maintain object identity across frames. This approach is computationally expensive and requires costly, identity-consistent annotations. The authors investigate whether the temporal consistency required for VIS can be achieved by strengthening the discriminative power of object queries during single-frame training, effectively bypassing the need for video-clip supervision.

The authors identify that standard object queries in models like Mask2Former drift across frames because they are only optimized for intra-frame localization. To address this, they introduce QueenVIS, which adds two auxiliary heads during training: a feature-prediction head that aligns queries with pooled backbone descriptors, and a center-prediction head that injects spatial structure. These heads are discarded at inference, ensuring zero additional parameters. For tracking, the authors implement a training-free association scheme that uses confidence-guided query propagation and a non-parametric memory bank to maintain identities across frames.

QueenVIS significantly outperforms the image-only baseline (MinVIS) across standard benchmarks, achieving a +6.7 AP improvement on YouTube-VIS 2021 and a +4.8 AP gain on OVIS. By enriching queries with appearance and spatial priors, the model reaches performance levels comparable to many video-supervised methods. The authors demonstrate that these gains are primarily due to improved cross-frame association, effectively closing the tracking-recall gap to near-oracle levels without processing a single video clip during training.