Hao Li, Ganlong Zhao, Yufei Liu, Haotian Hou, Guoquan Ye, Tongyan Fang, Chunxiao Liu, Siyuan Huang, Jianbo Liu, Xiaogang Wang, Hongsheng Li
5 min
Abstract
Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.
Sam: So it trusts its own experience more, but still uses the human video to fill in the gaps and see a wider variety of situations.
Alex: Exactly. The human data provides what you might call behavioral diversity — a broader range of situations and approaches — without overriding the precise, reliable data from actual robot demonstrations.
Sam: And this actually works in practice?
Alex: The paper suggests it does. By adding hundreds of hours of human video alongside a smaller set of real robot recordings, they saw a meaningful improvement in performance on physical tasks like scooping and pouring.
Sam: What about the weak points? Does it struggle when the view is blocked or the movement is too fast to track clearly?
Alex: You've identified a genuine limitation the authors acknowledge. Their hand-tracking pipeline can be prone to what they call "jitter" — where the tracked position shakes around unreliably — or "occlusion," where the hand disappears behind an object. In those moments, the robot is essentially learning from a blurry or incomplete signal.
Sam: So it can pick up general movement patterns, but very fine-grained or subtle manipulations are harder to capture.
Alex: That's right. The system handles broad positioning well, but the authors note that extremely precise, delicate movements remain a challenge when you're learning from video alone.
Sam: The paper mentions future work might bring in tactile feedback. How would that change things?
Alex: Right now, the system only "sees" — it learns from visual information. If it could also infer pressure, or how an object deforms when touched, it might be able to learn much more delicate tasks. Think of the difference between watching someone crack an egg and actually feeling how much force is needed. That's the gap tactile feedback would begin to close.
Sam: So this is a solid foundation, but there's still a meaningful distance between watching a video and physically understanding the world.
Alex: That's a fair way to put it. ACE-EGO-0 is a meaningful step toward scaling robot learning beyond the limits of traditional data collection — but the next frontier is moving from purely visual learning toward understanding physical contact. Thanks for listening to ResearchPod.