Jeongeun Park, Taekyung Kim, Dongyoon Han, Sangdoo Yun
9 min
Abstract
World-action models inherit broad visual and physical priors from video pre-training. However, standard within-video next-frame prediction allows them to rely on appearance shared between the context and future, rather than learning interactions that transfer across scenes and embodiments. We introduce WATCH, a cross-video pre-training that learns from task-paired human demonstrations collected across diverse environments. Conditioned on the video and actions of one demonstration, the model jointly predicts another demonstration's actions and future frames from its current observation. Predicting one demonstration from another encourages the model to learn task-relevant interactions that transfer across scenes and demonstrators. Our pre-training shows that these capabilities can be learned from readily scalable human-human pairs alone, without requiring matched human-robot demonstrations. Analyses before robot post-training show improved motion alignment across embodiments and better action and video prediction on unseen robots, with representations closer to those learned from robot pairs. After robot post-training, WATCH improves MimicDroid in-context success from 40.9% to 68.3% with Cosmos Policy and from 59.8% to 77.7% with Cosmos3, with the largest gains on the unseen levels. Gains persist under context-free post-training across robot embodiments. WATCH outperforms within-video pre-training and co-training baselines, supporting cross-video prediction as a scalable approach to learning transferable world-action representations.
Sam: They first restrict candidates using a task taxonomy, a hierarchy of activity labels. Then they rank candidates by motion similarity, allowing their timing to stretch for comparison. They emphasize motion shape over reach distance, so different-sized movements can still match.
Alex: That means the reference says what kind of movement to make, not necessarily how far to move.
Sam: The required extent has to come from the current image and state. Training randomly selects from each query's reference set, which can include demonstrations from its own source and other sources. The build contains roughly eleven million query–reference links.
Alex: Those are links, not eleven million independent demonstrations. And this isn't learning from arbitrary internet video.
Sam: Both distinctions matter. Windows and references are reused across links. The human data come from one corpus, EgoVerse, with task information and tracked hands. Extending the pairing to less structured sources is future work.
Alex: What's genuinely new relative to earlier human-video robot learning? Supplying a demonstration as guidance wasn't invented here.
Sam: Related systems already learn from task-matched human–robot demonstrations. WATCH uses human–human pairs during its proposed pre-training stage, the learning before target-robot adaptation. Its contribution is the cross-demonstration prediction task, not the invention of demonstration-guided control.
Alex: Let's separate learning better representations from actually controlling a robot. Where does the strongest practical evidence sit?
Sam: In the post-training comparisons. On MimicDroid, they adapt models to a GR1 humanoid using two hundred fifty-three robot training episodes. Those episodes come from augmented and retargeted human demonstrations, rather than a released collection of robot-executed trajectories.
Alex: So what does success mean here? Matching familiar scenes would be less persuasive than handling a changed setting.
Sam: The benchmark separates training settings, changed appearances and objects, and held-out layouts. During demonstration-guided evaluation, the robot receives a human demonstration from its test setting. WATCH improves all levels, with the largest gains under environment shift.
Alex: Does the result depend on the particular video backbone, the starting network underneath the policy?
Sam: They also test Cosmos3, a backbone already trained on human and robot action data. Its average MimicDroid success rises from about sixty percent to seventy-eight percent. So the proposed human-only stage can add value even to an already action-trained backbone.
Alex: The reference could still be doing most of the work. What happens when you remove it during robot training and evaluation?
Sam: Benefits persist. That's their context-free setting: predicting robot actions from the current observation, state, and instruction, without a demonstration. They find gains on MimicDroid, on bimanual ALOHA tasks in TableTop-Sim, and on Franka kitchen tasks in RoboCasa.
Alex: Are those improvements uniformly large, or does the summary hide a weaker case?
Sam: The Cosmos3 improvement on RoboCasa is small. And that particular setup adds robot data during pre-training, so it isn't a clean human-only example. The Cosmos Policy version shows a clearer gain there. The breadth is encouraging, but the benefits aren't uniform.
Alex: What supports their explanation that the model learns interactions, rather than simply benefiting from extra training?
Sam: Their ablations compare WATCH with within-video training and with co-training, which adds robot examples to within-video learning. WATCH performs better across the reported downstream comparisons. Predicting both actions and images also works better overall than supervising either alone.
Alex: That supports the training objective. What about their claim that representations become less tied to the body performing the movement?
Sam: Before robot adaptation, matching motions across people and robots become easier to retrieve from the model's internal features. Attention also shifts toward acting hands and manipulated objects. Those observations support the interpretation, but don't establish a complete causal explanation.
Alex: Is there any reason to treat those feature tests more cautiously than the control results?
Sam: Yes. In the motion analyses, WATCH also receives a matched reference's future actions, information the baselines don't receive. The authors acknowledge that this can contribute to the scores. So those tests don't isolate changes in the learned representation alone.
Alex: For someone considering an implementation, what's the limitation most likely to change whether this applies?
Sam: The shared action space. Human and robot movements are described through hand positions, orientations, and grasp information. All tested robot targets use Cartesian commands: where the hand should go, rather than how each individual joint should move.
Alex: So a policy commanding individual joints can't simply inherit the same action prior.
Sam: That case hasn't been demonstrated. It would need an intermediate hand-motion target or a learned mapping. Also, every training configuration is a single run, and the data-scaling experiment increases compute alongside data. My reading is promising transfer evidence, not a settled estimate of reliability or data efficiency.
Alex: Who should read the full document, and where should they spend their attention first?
Sam: Researchers building robot policies from human demonstrations should read the post-training experiments and the pre-training ablations first. Then read the cross-video method and limitations, especially the shared action space and pairing requirements. For representation researchers, the analysis protocol is essential because it explains the extra reference information.
Alex: And what should everyone else remember after the benchmark names have faded?
Sam: Predicting across demonstrations can teach more transferable interactions than predicting within them. Here, that helped robot control after adaptation; it did not remove the need for robot training.
Alex: Thanks for listening.