Ruiqi Shen, Chang Liu, Henghui Ding
4 min
Abstract
Modern Video Object Segmentation (VOS) involves tracking and segmenting user-specified targets. While recent approaches have achieved remarkable performance in single-target scenarios, extending them to multi-target settings typically involves replicating the single-target processing for each individual object, resulting in reduced frame rates (FPS) with unbounded latency as target count increases. Built upon Segment Anything 2 (SAM2), we propose SAM-MT, which addresses this by transforming the model into an interactive framework for real-time Multi-Target video segmentation. SAM-MT uses explicit queries to represent different individual targets, in parallel with a shared representation for global context. It employs decoupled masked attention to keep individual identities distinct from cross-target interference, and sparse memory for stable temporal evolution, along with specialized strategies for occlusion handling and overlap prevention. SAM-MT successfully decouples latency from the number of targets, achieving real-time speed on par with single-target baselines (>36 FPS for 10 targets) while maintaining SAM2's robust video segmentation performance.
Sam: Does this actually hold up in practice? What kind of speed does it achieve?
Alex: The results are notable. Where previous models show a clear drop in performance as the number of targets increases, this approach maintains a smooth frame rate — over 36 frames per second — even when tracking ten objects simultaneously. For context, video that feels fluid to the human eye needs roughly 24 to 30 frames per second, so staying above that threshold with ten targets is a meaningful result.
Sam: So they've genuinely decoupled the computational cost from the number of targets. Adding more objects doesn't slow the system down in the way it used to.
Alex: That is the central finding. And there's one more design element worth mentioning — the paper describes something called an "overlap loss." It's essentially a penalty the system applies during training whenever it thinks a single pixel belongs to two different objects at once.
Sam: So it forces the system to make a definitive choice?
Alex: Exactly. Each pixel gets assigned to one target only. That rule, baked in during training, helps prevent the kind of confusion you'd otherwise see in crowded scenes — where two people are close together and the system starts blending their outlines.
Sam: What strikes me about all of this is that the solution wasn't simply "use more computing power." It was a more careful way of organizing what information gets shared, what gets kept separate, and what gets remembered.
Alex: That's a pattern you see often in this kind of research. Raw speed matters, but the bigger gains tend to come from rethinking the structure — deciding what the system actually needs to look at, and what it can safely ignore. SAM-MT is a clear example of that principle in practice. Thanks for listening to ResearchPod.