Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io .
Alex: Welcome to another episode of ResearchPod. Today, Sam, we're diving into a preprint called "Can Vision-Language Models Solve the Shell Game?" by Tiedong Liu and Wee Sun Lee from the National University of Singapore.
Sam: That's right, Alex. The paper tackles a simple puzzle: why can't advanced AI systems that watch videos and answer questions about them—like figuring out where a hidden ball ends up after some cups shuffle around—solve what's called the shell game, a task humans handle without thinking twice.
Alex: So this is basically asking if these AI video watchers have a blind spot in tracking moving objects over time? And the shell game is that test?
Sam: Exactly. Humans intuitively follow objects through motion from one video frame to the next, even if the objects look identical. But the paper shows current vision-language models—AI that combines seeing videos with understanding language—fail badly at this, guessing randomly on cleaned-up tests where looks don't give hints away.
Alex: Right, so they rely on easy visual tricks instead of real tracking?
Sam: Yes. The researchers audited benchmarks like the Perception Test, where real videos of cup shuffles often have clues like different cup colors or transparency. Strip those out, and top models like Gemini-3-Pro drop from 80% accuracy to around 30% on three-cup shuffles—no better than chance. To fix this gap, they created VET-Bench.
Alex: VET-Bench sounds like the real test here—what makes it different?
Sam: It's a set of synthetic videos with perfectly identical cups or cards that swap positions, forcing the AI to track solely by how things move continuously through space and time. No shortcuts. State-of-the-art models still perform near random, around one-third correct for three items, exposing they don't maintain object identities frame by frame.
Alex: Huh. So even fancy reasoning models can't follow the swaps properly.
Sam: Precisely. The paper argues this reveals a core weakness in how these transformer-based AIs process videos—they're limited without step-by-step tracking, much like struggling to follow a ball under identical cups without peeking. Their solution, called Spatiotemporal Grounded Chain-of-Thought, teaches the AI to explicitly map out object paths first.
Alex: Okay, so SGCoT maps out those paths explicitly. But what makes the base models so fundamentally bad at this without it—like, is there a built-in limit in how these AI architectures handle sequences of changes?
Sam: Yes, the paper provides a theoretical proof. Tracking identical objects through swaps is like composing a long chain of small position flips—each swap changes which object is where, and you need to track the overall effect step by step across many frames. Fixed-depth transformers, the core engine in these models, can't compose arbitrary numbers of those flips without intermediate steps; they hit an expressivity limit for this kind of sequential state-tracking problem.
Alex: Huh. So even in theory, without something like step-by-step notes, they can't keep up as swaps pile on?
Sam: Exactly. On tests with more swaps or objects, performance drops right to random guessing after just one swap—models that ace zero-swap cases, where nothing moves, collapse because they can't accumulate the changes reliably.
Alex: That lines up with what you said about audits like the Perception Test—do those real videos hide the same issue?
Sam: They do. When researchers filter to identical opaque cups with swaps, top models like Gemini-3-Pro fall from 80% on the full set to about 31%, matching random on their synthetic tests—no better than picking blindly.
Alex: Right, so they're exploiting easy cues, not truly following motion. What do their wrong answers look like up close?
Sam: Common patterns emerge: some guess randomly without reasoning; others describe swaps vaguely, like "the left cup moves right early on," but mix up which is which; and many hallucinate events, inventing swaps that never happened despite seeing the video correctly at first. Each error in early perception snowballs, since later logic builds on it—symbolic steps can't fix bad initial grounding.
Alex: So SGCoT sidesteps that by forcing precise path logs from the start.
Sam: Precisely. It uses timestamped coordinates—like GPS points every half-second—to create unambiguous updates per frame, avoiding vague time ranges or loose labels that cause drift in other chains of thought. This fine-grained grounding lets the model reason reliably to 91% accuracy after brief text tuning, a clear improvement over base near-random performance.
Alex: So that GPS-like logging per frame gives the model the step-by-step notes it needs to beat the theory limit you mentioned.
Sam: Yes. Without those explicit paths—listing each object's position and speed at exact times—the transformer layers can't stack up the changes from many swaps. By outputting structured tracks first, like a trail of breadcrumbs across frames, SGCoT supplies the intermediate computation that fixed-depth models lack, turning vague guesses into reliable final answers.
Alex: Okay, that clicks—it's outsourcing the hard sequencing to text steps. But does the paper talk about where this still falls short in messier real videos?
Sam: It does note limitations. The setup assumes clear swaps with no overlaps or blurs, where positions stay unambiguous under motion rules—like objects can't teleport or vanish. In everyday videos, things get tricky with crowding, occlusions where one hides another, or fuzzy motion; those might need extra physics knowledge or world models to track properly.
Alex: Huh, fair—so it's a strong fix for clean tracking, but real chaos adds layers.
Sam: Exactly. The paper positions VET-Bench as honing in on spatiotemporal gaps that other video QA tests miss, often shortcutted by static looks. Ultimately, the evidence suggests SGCoT is a meaningful advance for grounding reasoning in time and space.
Alex: Right, identifying that tracking bottleneck and bridging it with explicit paths. Makes sense why humans do it effortlessly—we build those mental trails intuitively. Thanks, Sam, for laying it out so clearly. That's our look at vision models and the shell game.