From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence

How can we bridge the embodiment gap between human manipulation videos and robot hardware to create scalable, executable training data for embodied AI? The author argues that the bottleneck in robot learning is not model architecture, but the lack of diverse, high-quality data that accounts for the differences in morphology and kinematics between humans and robots.

Pegasus moves away from raw video-to-video generation, which often fails to produce physically valid robot motions. Instead, it uses a structured, graph-based pipeline:

1. Task Graph Extraction: Human videos are parsed into semantic graphs representing sub-goals, objects, and spatial relationships.
2. Hierarchical Affordance Latent: A latent space organizes these tasks by object states and affordances rather than object identities, enabling generalization to unseen objects.
3. Graph Translation: The task graph is transformed into a robot-specific planning graph that incorporates kinematic and workspace constraints.
4. Closed-Loop Physics Verification: A verification module iteratively checks generated trajectories for collisions, joint limits, and inverse kinematics, providing feedback to refine the output until it is physically executable.

Pegasus demonstrates that structured knowledge transfer is a viable alternative to expensive real-robot data collection. Key results include:

• Improved Performance: Graph-conditioned generation outperforms prompt-only methods by 12.3% in task correctness and 15.7% in executability.
• Policy Learning: Augmenting real-robot data with Pegasus-generated demonstrations improves downstream policy success rates by 11.3%.
• Generalization: The affordance-based latent space achieves 78.5% task correctness on unseen objects, significantly outperforming object-identity-based models.
• Physical Validity: The closed-loop verification mechanism reduces kinematic errors by 93.5% and collision rates by 90.9%, ensuring that 94.1% of generated trajectories are executable.