World-action models inherit broad visual and physical priors from video pre-training. However, standard within-video next-frame prediction allows them to rely on appearance shared between the context and future, rather than learning interactions that transfer across scenes and embodiments. We introduce WATCH, a cross-video pre-training that learns from task-paired human demonstrations collected across diverse environments. Conditioned on the video and actions of one demonstration, the model jointly predicts another demonstration's actions and future frames from its current observation. Predicting one demonstration from another encourages the model to learn task-relevant interactions that transfer across scenes and demonstrators. Our pre-training shows that these capabilities can be learned from readily scalable human-human pairs alone, without requiring matched human-robot demonstrations. Analyses before robot post-training show improved motion alignment across embodiments and better action and video prediction on unseen robots, with representations closer to those learned from robot pairs. After robot post-training, WATCH improves MimicDroid in-context success from 40.9% to 68.3% with Cosmos Policy and from 59.8% to 77.7% with Cosmos3, with the largest gains on the unseen levels. Gains persist under context-free post-training across robot embodiments. WATCH outperforms within-video pre-training and co-training baselines, supporting cross-video prediction as a scalable approach to learning transferable world-action representations.
Sam: Robot models can learn more useful skills by predicting across different people's demonstrations. In this study, that extra training improved robot control after adaptation, even when no demonstration was supplied during use.
Alex: That sounds useful if collecting robot experience is the bottleneck. How much did it change performance?
Sam: On the MimicDroid benchmark, one model's average success rose from about forty-one percent to sixty-eight percent. That's from a method preprint by Jeongeun Park and colleagues at NAVER AI Lab and Georgia Tech. They call their approach WATCH.
Alex: So the audience isn't just people building video generators. It's researchers trying to turn plentiful human demonstrations into useful robot training.
Sam: And the distinction matters: the proposed training stage needs human pairs, not matched human–robot demonstrations. Robots still need adaptation afterward. We'll unpack what changes in the learning task, what supports the transfer claim, and where the evidence stops.
Alex: Start with the usual learning task. What can a model get wrong while apparently predicting video well?
Sam: The authors study world-action models: models that predict movements and their visual consequences together. They build on video models trained to predict how scenes evolve. Usually, the observation and the future being predicted come from the same demonstration.
Alex: So the room, camera, and person remain consistent. A model could use that continuity without separating the interaction from its surroundings.
Sam: That's the authors' concern. Within-video prediction doesn't explicitly require similar movements to have consistent representations across different bodies and scenes. They hypothesize that this makes later robot adaptation harder.
Alex: What does WATCH make the model do instead? Give me a concrete pair rather than the architecture.
Sam: Take unloading a dishwasher. The document pairs a demonstration from one collection source with another person's demonstration from a different source. The model receives the reference person's video and movements, plus the other person's current image and body state.
Alex: And the target isn't to reproduce the reference video. It's to predict what happens in the other person's scene.
Sam: It predicts that person's next movements and future images. The reference supplies an example of task progression. The current observation supplies the setting where that interaction must happen. Different appearances make direct visual copying less useful.
Alex: But “unloading a dishwasher” could contain quite different movements. How do they avoid pairing demonstrations that share only a label?
Sam: They first restrict candidates using a task taxonomy, a hierarchy of activity labels. Then they rank candidates by motion similarity, allowing their timing to stretch for comparison. They emphasize motion shape over reach distance, so different-sized movements can still match.
Alex: That means the reference says what kind of movement to make, not necessarily how far to move.
Sam: The required extent has to come from the current image and state. Training randomly selects from each query's reference set, which can include demonstrations from its own source and other sources. The build contains roughly eleven million query–reference links.
Alex: Those are links, not eleven million independent demonstrations. And this isn't learning from arbitrary internet video.
Sam: Both distinctions matter. Windows and references are reused across links. The human data come from one corpus, EgoVerse, with task information and tracked hands. Extending the pairing to less structured sources is future work.
Alex: What's genuinely new relative to earlier human-video robot learning? Supplying a demonstration as guidance wasn't invented here.
Sam: Related systems already learn from task-matched human–robot demonstrations. WATCH uses human–human pairs during its proposed pre-training stage, the learning before target-robot adaptation. Its contribution is the cross-demonstration prediction task, not the invention of demonstration-guided control.
Alex: Let's separate learning better representations from actually controlling a robot. Where does the strongest practical evidence sit?
Sam: In the post-training comparisons. On MimicDroid, they adapt models to a GR1 humanoid using two hundred fifty-three robot training episodes. Those episodes come from augmented and retargeted human demonstrations, rather than a released collection of robot-executed trajectories.
Alex: So what does success mean here? Matching familiar scenes would be less persuasive than handling a changed setting.
Sam: The benchmark separates training settings, changed appearances and objects, and held-out layouts. During demonstration-guided evaluation, the robot receives a human demonstration from its test setting. WATCH improves all levels, with the largest gains under environment shift.
Alex: Does the result depend on the particular video backbone, the starting network underneath the policy?
Sam: They also test Cosmos3, a backbone already trained on human and robot action data. Its average MimicDroid success rises from about sixty percent to seventy-eight percent. So the proposed human-only stage can add value even to an already action-trained backbone.
Alex: The reference could still be doing most of the work. What happens when you remove it during robot training and evaluation?
Sam: Benefits persist. That's their context-free setting: predicting robot actions from the current observation, state, and instruction, without a demonstration. They find gains on MimicDroid, on bimanual ALOHA tasks in TableTop-Sim, and on Franka kitchen tasks in RoboCasa.
Alex: Are those improvements uniformly large, or does the summary hide a weaker case?
Sam: The Cosmos3 improvement on RoboCasa is small. And that particular setup adds robot data during pre-training, so it isn't a clean human-only example. The Cosmos Policy version shows a clearer gain there. The breadth is encouraging, but the benefits aren't uniform.
Alex: What supports their explanation that the model learns interactions, rather than simply benefiting from extra training?
Sam: Their ablations compare WATCH with within-video training and with co-training, which adds robot examples to within-video learning. WATCH performs better across the reported downstream comparisons. Predicting both actions and images also works better overall than supervising either alone.
Alex: That supports the training objective. What about their claim that representations become less tied to the body performing the movement?
Sam: Before robot adaptation, matching motions across people and robots become easier to retrieve from the model's internal features. Attention also shifts toward acting hands and manipulated objects. Those observations support the interpretation, but don't establish a complete causal explanation.
Alex: Is there any reason to treat those feature tests more cautiously than the control results?
Sam: Yes. In the motion analyses, WATCH also receives a matched reference's future actions, information the baselines don't receive. The authors acknowledge that this can contribute to the scores. So those tests don't isolate changes in the learned representation alone.
Alex: For someone considering an implementation, what's the limitation most likely to change whether this applies?
Sam: The shared action space. Human and robot movements are described through hand positions, orientations, and grasp information. All tested robot targets use Cartesian commands: where the hand should go, rather than how each individual joint should move.
Alex: So a policy commanding individual joints can't simply inherit the same action prior.
Sam: That case hasn't been demonstrated. It would need an intermediate hand-motion target or a learned mapping. Also, every training configuration is a single run, and the data-scaling experiment increases compute alongside data. My reading is promising transfer evidence, not a settled estimate of reliability or data efficiency.
Alex: Who should read the full document, and where should they spend their attention first?
Sam: Researchers building robot policies from human demonstrations should read the post-training experiments and the pre-training ablations first. Then read the cross-video method and limitations, especially the shared action space and pairing requirements. For representation researchers, the analysis protocol is essential because it explains the extra reference information.
Alex: And what should everyone else remember after the benchmark names have faded?
Sam: Predicting across demonstrations can teach more transferable interactions than predicting within them. Here, that helped robot control after adaptation; it did not remove the need for robot training.
Alex: Thanks for listening.