WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation

As robots increasingly work alongside humans, they must move beyond static scene understanding to interpret human intent, procedural history, and spatial context. Existing benchmarks typically provide a single image or a demonstration to mimic, failing to test a robot's ability to reason about observed human activity. The authors introduce WatchAct to evaluate how well robots can ground language instructions in the observed behavior of another agent.

WatchAct consists of 3,000 long-horizon instances across 14 tasks, categorized into four cognitive domains: Event Grounding, Procedural Reasoning, Implicit Intent Inference, and Episodic Reasoning. Each instance pairs a real-world video of human activity with an aligned simulator task in LIBERO. The authors propose a disentangled evaluation protocol that separates performance into three stages: video-to-plan reasoning (using vision-language models), policy execution (using oracle plans), and full integrated pipeline completion. This allows researchers to pinpoint whether failures stem from poor high-level planning or inadequate low-level motor control.

Current systems are far from solving the tasks in WatchAct. The best-performing pipeline, combining Gemini-3.1-Pro with the pi-0.5 policy, achieved only a 16.3% success rate in simulation and 14.0% on a real Franka Research 3 robot. The authors identified a massive gap in planning: while humans achieved a 97.1% success rate in generating correct plans from video, the best model reached only 36.8%. Furthermore, even when provided with perfect oracle plans, robotic policies struggled to execute the required physical actions, highlighting that both high-level reasoning and low-level control remain significant bottlenecks.

WatchAct provides a rigorous framework for evaluating embodied AI in human-centric environments. By isolating the components of the perception-to-action chain, it demonstrates that current models lack the necessary cognitive depth to interpret human actions, suggesting that future research must focus on better temporal and spatial reasoning over observed human behavior rather than just static image-to-action mapping.