UAV-OVVIS: Unmanned Aerial Vehicles Also Need Open-Vocabulary Video Instance Segmentation

Unmanned Aerial Vehicle (UAV) video analysis is critical for tasks like traffic monitoring and urban management. Traditionally, these systems rely on closed-set, box-level detection, which limits their ability to adapt to new categories or provide precise, pixel-level instance segmentation. As visual perception shifts toward open-vocabulary understanding, there is a growing need for systems that can identify and track arbitrary objects specified by text queries in dynamic, top-down aerial views.

To address these needs, the authors introduce the UAV-OVVIS task and the AeroTrack framework. AeroTrack is a training-free, modular system that decouples the perception process into two primary components:

1. Open-Vocabulary Recognizer: Periodically performs detection on key frames using models like YOLO-World or Grounding DINO to generate positional prompts.
2. Video Instance Segmenter: Uses these prompts to generate and propagate masks within short temporal segments (e.g., using SAM2 or SAM3).

To manage the computational costs and memory constraints of long UAV videos, the system resets the segmenter's internal state at fixed intervals. To ensure the resulting trajectories remain consistent, the authors developed a Lifecycle-aware ID Association (LIA) module. LIA operates at the output level to link local instance IDs across segments, effectively maintaining global identity without requiring additional training.

Because existing datasets lack the instance-level annotations required for this task, the authors constructed AeroVIS. This benchmark includes 9 distinct UAV object categories and over 8,000 trajectories. Experimental results demonstrate that AeroTrack significantly outperforms traditional video instance segmentation methods in UAV scenarios, proving that leveraging existing foundation models is a viable, robust strategy for open-vocabulary aerial perception.

UAV-OVVIS

A task requiring the discovery and segmentation of targets based on open-vocabulary text queries, resulting in globally consistent instance trajectories.

Training-free

An approach that utilizes pre-trained foundation models directly for inference without requiring domain-specific fine-tuning or weight updates.

Lifecycle-aware ID Association (LIA)

A post-processing module that resolves identity fragmentation by linking local instance masks across temporal segments based on geometric consistency.

Positional Prompt

Spatial information (such as boxes or points) derived from an open-vocabulary detector that guides the segmenter to focus on specific targets.

Periodic Refreshing

The strategy of re-detecting objects at fixed intervals to prevent error accumulation and manage memory usage in long video sequences.

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.