Dingyi Rong, Yue Shi, Chaofan Ma, Jiezhang Cao, Zongrui Wang, Zeyu Zhang, Yao Mu, Guangtao Zhai, Ning Liu
6 min
Abstract
Large-scale manipulation data is essential for robot learning, yet collecting robot demonstrations remains expensive and difficult to scale. Meanwhile, abundant egocentric human manipulation videos provide rich behavioral experiences, but transferring them across embodiments remains challenging due to differences between human hands and robotic end-effectors. Recent advances in video world models offer a promising pathway to synthesize robot-centric manipulation videos from human observations, while their cross-embodiment transfer capability remains largely unexplored. Therefore, we introduce H2R-Bench, a benchmark for evaluating cross-embodiment human-to-robot manipulation video generation, where models transform egocentric human demonstrations into robot manipulation videos under specified embodiments. Each benchmark instance contains a human demonstration video, target embodiment constraints, and source-grounded annotations covering task goals, action events, functional contacts, and object responses. H2R-Bench evaluates generated videos through five dimensions, including goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and general video quality. We benchmark eleven state-of-the-art video generation models across six manipulation families and two robot embodiments. Our evaluation reveals that current video world models remain limited in human-to-robot manipulation transfer: even leading models often fail in embodiment consistency, functional interaction, and task execution. H2R-Bench provides a systematic diagnostic framework for evaluating whether video world models can bridge the human-to-robot embodiment gap and convert human manipulation observations into robot-centric training resources.
Sam: Exactly. Think of it like a master chef teaching an apprentice who wears prosthetic gripping tools. The apprentice doesn't need to copy every single finger movement the chef makes. But they do need to hold the spatula firmly enough to actually flip the pancake. The benchmark is checking for that firm grip — not just whether the apprentice looks like a chef.
Alex: That's a useful way to frame it. So what did the tests actually reveal?
Sam: They ran eleven different video generation models through two hundred forty transfer cases. The models fell into two broad groups: systems that take a full continuous video as input, and systems that receive a set of individual still frames. The full-video systems consistently outperformed the frame-based ones.
Alex: Why would watching a continuous video make such a difference compared to a set of separate images?
Sam: Because motion over time carries information that static snapshots simply can't capture — things like velocity, acceleration, and the precise moment a tool makes contact with an object. When a model processes a continuous stream, it builds an internal sense of how objects move and interact. When it only sees disconnected frames, it loses those cues and frequently defaults to rendering a human hand rather than successfully substituting the robot body.
Alex: So the motion itself is carrying the instructions, not just the starting and ending pictures.
Sam: That's a good way to put it. And this is precisely why the benchmark captures such a clear failure mode: when you replace a full video with a sparse set of still frames, performance drops significantly across the board. The model can no longer track the physical interaction through time, so it can't reliably determine how the robot's end-effector — the tool or gripper at the tip of the arm — should make contact with the object.
Alex: Did giving the models a reference image of the target robot help close that gap?
Sam: The results there were genuinely mixed, which is itself an interesting finding. For one model, Wan 2.7, adding a reference image of the target robot improved its scores noticeably. But for two other top-performing models, adding that same reference image actually caused scores to drop.
Alex: So more information isn't always helpful if the model doesn't know how to use it properly.
Sam: Precisely. It suggests these models don't yet have a reliable internal understanding of robot body structure. Giving them a picture of the robot sometimes confuses them rather than helping. The overall pattern across all eleven models was the same: strong visual rendering, but persistent struggles with embodiment consistency and physical contact accuracy.
Alex: Which brings us to what the paper says about the limitations of this approach.
Sam: The authors are candid about this. The evaluation relies on automated judgments and sampled visual evidence, which means tricky camera angles, objects blocking the view, or blurry frames can create blind spots in the scoring. The system can't always tell whether contact happened correctly if the key moment is obscured.
Alex: So even with a careful benchmark, there are still situations where the automated judge isn't sure what it's seeing.
Sam: Right. That's why the authors point toward future work involving denser visual tracking and more detailed three-dimensional reconstruction — tools that could give a more precise picture of what's physically happening, rather than relying on what's visible from a single camera angle.
Alex: It sounds like H2R-Bench is a meaningful step forward for diagnosing these problems, but not the final word.
Sam: That's a fair characterization. The paper's contribution is largely diagnostic — it gives the field a clearer language for talking about why video models fail at physical transfer, not just that they fail. And that distinction matters, because you can't fix a problem you haven't properly measured.
Alex: So the honest takeaway is that video generation has gotten quite good at producing convincing footage, but turning human behavior captured on video into reliable physical instructions for a robot is still largely unsolved — and this benchmark is a tool for understanding exactly where the gaps are.
Sam: That's the sober reality. The gap between visual plausibility and physical reliability is real, it's measurable, and closing it will require models that genuinely understand the mechanics of contact and motion — not just models that know what a task is supposed to look like.
Alex: Thanks for walking us through it. And thanks to everyone listening to ResearchPod.