Large-scale manipulation data is essential for robot learning, yet collecting robot demonstrations remains expensive and difficult to scale. Meanwhile, abundant egocentric human manipulation videos provide rich behavioral experiences, but transferring them across embodiments remains challenging due to differences between human hands and robotic end-effectors. Recent advances in video world models offer a promising pathway to synthesize robot-centric manipulation videos from human observations, while their cross-embodiment transfer capability remains largely unexplored. Therefore, we introduce H2R-Bench, a benchmark for evaluating cross-embodiment human-to-robot manipulation video generation, where models transform egocentric human demonstrations into robot manipulation videos under specified embodiments. Each benchmark instance contains a human demonstration video, target embodiment constraints, and source-grounded annotations covering task goals, action events, functional contacts, and object responses. H2R-Bench evaluates generated videos through five dimensions, including goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and general video quality. We benchmark eleven state-of-the-art video generation models across six manipulation families and two robot embodiments. Our evaluation reveals that current video world models remain limited in human-to-robot manipulation transfer: even leading models often fail in embodiment consistency, functional interaction, and task execution. H2R-Bench provides a systematic diagnostic framework for evaluating whether video world models can bridge the human-to-robot embodiment gap and convert human manipulation observations into robot-centric training resources.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a notable study on how video generation models handle the translation from human demonstrations to robotic execution.
Alex: So the core question is: can we take ordinary videos of people doing everyday tasks and use them to teach robots — without needing expensive, specialized equipment?
Sam: That's exactly it. We have enormous amounts of first-person video footage of humans doing things — cooking, assembling objects, picking things up. The challenge is that a robot has a completely different body. Its "hands" might be a simple two-jaw clamp or a multi-jointed mechanical arm. Teaching it to copy human actions from video is a major unsolved problem.
Alex: Right, and that physical mismatch between human hands and robot grippers is what makes this so tricky.
Sam: Researchers call it the embodiment gap. Video world models — systems that can generate or predict realistic video sequences — seemed like a promising tool for bridging that gap. But nobody had seriously tested whether a video that looks realistic actually translates into a robot doing the task correctly.
Alex: So looking good visually doesn't necessarily mean the robot is succeeding at the task?
Sam: That's the central finding. The authors built an evaluation framework called H2R-Bench to test this directly. And what they found is that high visual quality and functional physical success are surprisingly disconnected. A model can render a smooth, convincing scene while completely failing at the actual physical interaction.
Alex: I would have assumed that if the video looks realistic, the robot actions inside it would make sense too.
Sam: It's a reasonable assumption, but it doesn't hold up. Think of it like a movie stunt. A film can show someone catching a falling vase in a way that looks completely natural — but if you tried to use that footage as instructions for a robot arm, the robot might miss entirely, because the video was optimized for appearance, not for physics.
Alex: So how does H2R-Bench actually test whether the physics are right?
Sam: Instead of just asking "does this look good?", it breaks the evaluation into five separate checks. Did the robot reach the task goal? Did it perform the right sequence of actions? Did it make physically sensible contact with the object? Did it use the correct robot body — not accidentally render a human hand? And only then: how does the overall video quality look?
Alex: So it's essentially a checklist that separates appearance from function.
Sam: Exactly. Think of it like a master chef teaching an apprentice who wears prosthetic gripping tools. The apprentice doesn't need to copy every single finger movement the chef makes. But they do need to hold the spatula firmly enough to actually flip the pancake. The benchmark is checking for that firm grip — not just whether the apprentice looks like a chef.
Alex: That's a useful way to frame it. So what did the tests actually reveal?
Sam: They ran eleven different video generation models through two hundred forty transfer cases. The models fell into two broad groups: systems that take a full continuous video as input, and systems that receive a set of individual still frames. The full-video systems consistently outperformed the frame-based ones.
Alex: Why would watching a continuous video make such a difference compared to a set of separate images?
Sam: Because motion over time carries information that static snapshots simply can't capture — things like velocity, acceleration, and the precise moment a tool makes contact with an object. When a model processes a continuous stream, it builds an internal sense of how objects move and interact. When it only sees disconnected frames, it loses those cues and frequently defaults to rendering a human hand rather than successfully substituting the robot body.
Alex: So the motion itself is carrying the instructions, not just the starting and ending pictures.
Sam: That's a good way to put it. And this is precisely why the benchmark captures such a clear failure mode: when you replace a full video with a sparse set of still frames, performance drops significantly across the board. The model can no longer track the physical interaction through time, so it can't reliably determine how the robot's end-effector — the tool or gripper at the tip of the arm — should make contact with the object.
Alex: Did giving the models a reference image of the target robot help close that gap?
Sam: The results there were genuinely mixed, which is itself an interesting finding. For one model, Wan 2.7, adding a reference image of the target robot improved its scores noticeably. But for two other top-performing models, adding that same reference image actually caused scores to drop.
Alex: So more information isn't always helpful if the model doesn't know how to use it properly.
Sam: Precisely. It suggests these models don't yet have a reliable internal understanding of robot body structure. Giving them a picture of the robot sometimes confuses them rather than helping. The overall pattern across all eleven models was the same: strong visual rendering, but persistent struggles with embodiment consistency and physical contact accuracy.
Alex: Which brings us to what the paper says about the limitations of this approach.
Sam: The authors are candid about this. The evaluation relies on automated judgments and sampled visual evidence, which means tricky camera angles, objects blocking the view, or blurry frames can create blind spots in the scoring. The system can't always tell whether contact happened correctly if the key moment is obscured.
Alex: So even with a careful benchmark, there are still situations where the automated judge isn't sure what it's seeing.
Sam: Right. That's why the authors point toward future work involving denser visual tracking and more detailed three-dimensional reconstruction — tools that could give a more precise picture of what's physically happening, rather than relying on what's visible from a single camera angle.
Alex: It sounds like H2R-Bench is a meaningful step forward for diagnosing these problems, but not the final word.
Sam: That's a fair characterization. The paper's contribution is largely diagnostic — it gives the field a clearer language for talking about why video models fail at physical transfer, not just that they fail. And that distinction matters, because you can't fix a problem you haven't properly measured.
Alex: So the honest takeaway is that video generation has gotten quite good at producing convincing footage, but turning human behavior captured on video into reliable physical instructions for a robot is still largely unsolved — and this benchmark is a tool for understanding exactly where the gaps are.
Sam: That's the sober reality. The gap between visual plausibility and physical reliability is real, it's measurable, and closing it will require models that genuinely understand the mechanics of contact and motion — not just models that know what a task is supposed to look like.
Alex: Thanks for walking us through it. And thanks to everyone listening to ResearchPod.