Ye Wang, Pei Lin, Xiong-Hui Chen, Haoqi Yuan, Zhixuan Liang, Yiyang Huang, Anzhe Chen, Zixing Lei, Jie Zhang, Tao Zhang, Haoyang Li, Tong Zhang, Chenxi Xiao, Ziyuan Jiao, Qin Jin
6 min
Abstract
Learning generalizable robot manipulation policies requires large-scale and diverse demonstration data. Egocentric human manipulation videos offer rich scene and task diversity, and prior work has shown that retargeting and rendering such videos into robot-format data can yield effective per-task policies at small scale. However, whether this approach can provide pretraining benefits for vision-language-action models at scale remains unexplored. We present \textbf{Ego2Robot}, a scalable pipeline that converts egocentric human manipulation videos into robot training data through action retargeting, robot-arm visual synthesis, and multi-level quality curation. Ego2Robot supports both curated datasets and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies, making it the largest ego-to-robot dataset to date. To evaluate generalization, we extend RoboTwin2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics. Experiments show that joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits validated on real-robot deployment. Project page: https://www-ye.github.io/ego2robot_blog/
Alex: So it's erasing the human and painting in a robot, frame by frame. That process, applied to thousands of hours of footage, must produce an enormous amount of training material.
Sam: It does. The researchers generated training data covering fifteen different robot types—enough to train what are called vision-language-action models. These are AI systems that look at a scene, process what they're seeing, and translate that into physical actions. The scale here is notable: it's the largest dataset of this kind produced to date.
Alex: And does all of that data actually help the robots handle situations they haven't seen before?
Sam: That's what the evaluation was designed to test. Rather than giving the robot one overall score, the researchers separated performance into four independent dimensions: how the scene looks visually, how the objects are arranged, what shape the robot itself is, and what the actual task requires. They called this a disentangled evaluation.
Alex: Why go to the trouble of separating those out?
Sam: Because a single score can hide a lot. If a robot fails, you want to know whether it was confused by the lighting, thrown off by an unfamiliar object position, struggling because it was a different robot model than it trained on, or simply didn't understand the task. By isolating each factor, the researchers found that training on the synthetic data made robots meaningfully more robust—particularly when the visual environment or the robot's own physical form differed from what they'd seen in training.
Alex: And what are the limits of this approach? Where does it fall short?
Sam: The most significant limitation is what happens with the hands. Because the pipeline maps all human hand movements onto a simple two-paddle gripper, it loses the nuance of how we actually use our fingers. Think about picking up a needle, or rotating a coin between your thumb and forefinger—those actions rely on independent finger control that a standard gripper simply can't replicate. The researchers are clear that this is a starting point for general manipulation tasks, not a solution for fine-grained, dexterous work.
Alex: It's a bit like teaching someone to paint by only giving them a wide house-painting brush. You can cover a lot of ground, but the fine detail work isn't there yet.
Sam: That's a fair way to put it. There's also the question of data quality. Because the system learns from whatever the human in the video does, it inherits human habits—including inefficient ones. The pipeline includes a filtering step to catch the worst of this, but the robot is still, in a sense, learning from human behavior, which isn't always the most efficient approach for a machine.
Alex: So it inherits both our skills and our bad habits.
Sam: To some degree, yes. The methodology is designed to be robust, but it's still shaped by the quality of the source material. That said, the ability to generate this volume and variety of training data is a clear improvement over earlier methods that depended on small, hand-curated datasets collected with physical robots. The core argument of the paper is that we don't always need more hardware—sometimes, we just need smarter ways to learn from the vast amount of human experience already captured on video.
Alex: That's a useful way to frame it. Thanks for walking through this, Sam.
Sam: It's been a pleasure. Thanks for listening to ResearchPod.