Tiedong Liu, Wee Sun Lee
6 min
Abstract
Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io .
Alex: Huh. So even in theory, without something like step-by-step notes, they can't keep up as swaps pile on?
Sam: Exactly. On tests with more swaps or objects, performance drops right to random guessing after just one swap—models that ace zero-swap cases, where nothing moves, collapse because they can't accumulate the changes reliably.
Alex: That lines up with what you said about audits like the Perception Test—do those real videos hide the same issue?
Sam: They do. When researchers filter to identical opaque cups with swaps, top models like Gemini-3-Pro fall from 80% on the full set to about 31%, matching random on their synthetic tests—no better than picking blindly.
Alex: Right, so they're exploiting easy cues, not truly following motion. What do their wrong answers look like up close?
Sam: Common patterns emerge: some guess randomly without reasoning; others describe swaps vaguely, like "the left cup moves right early on," but mix up which is which; and many hallucinate events, inventing swaps that never happened despite seeing the video correctly at first. Each error in early perception snowballs, since later logic builds on it—symbolic steps can't fix bad initial grounding.
Alex: So SGCoT sidesteps that by forcing precise path logs from the start.
Sam: Precisely. It uses timestamped coordinates—like GPS points every half-second—to create unambiguous updates per frame, avoiding vague time ranges or loose labels that cause drift in other chains of thought. This fine-grained grounding lets the model reason reliably to 91% accuracy after brief text tuning, a clear improvement over base near-random performance.
Alex: So that GPS-like logging per frame gives the model the step-by-step notes it needs to beat the theory limit you mentioned.
Sam: Yes. Without those explicit paths—listing each object's position and speed at exact times—the transformer layers can't stack up the changes from many swaps. By outputting structured tracks first, like a trail of breadcrumbs across frames, SGCoT supplies the intermediate computation that fixed-depth models lack, turning vague guesses into reliable final answers.
Alex: Okay, that clicks—it's outsourcing the hard sequencing to text steps. But does the paper talk about where this still falls short in messier real videos?
Sam: It does note limitations. The setup assumes clear swaps with no overlaps or blurs, where positions stay unambiguous under motion rules—like objects can't teleport or vanish. In everyday videos, things get tricky with crowding, occlusions where one hides another, or fuzzy motion; those might need extra physics knowledge or world models to track properly.
Alex: Huh, fair—so it's a strong fix for clean tracking, but real chaos adds layers.
Sam: Exactly. The paper positions VET-Bench as honing in on spatiotemporal gaps that other video QA tests miss, often shortcutted by static looks. Ultimately, the evidence suggests SGCoT is a meaningful advance for grounding reasoning in time and space.
Alex: Right, identifying that tracking bottleneck and bridging it with explicit paths. Makes sense why humans do it effortlessly—we build those mental trails intuitively. Thanks, Sam, for laying it out so clearly. That's our look at vision models and the shell game.