DreamX Team
4 min
How can video world models for robotic manipulation predict future observations that are not only photorealistic but also strictly faithful to commanded robot actions and physically consistent with object interactions?
To ensure that generated rollouts respect physical constraints and arm identities without drifting or losing objects, the authors introduce DreamX-Phi 1.0, built on top of the Wan2.2-TI2V-5B video diffusion transformer. The approach incorporates several key innovations: an arm-grouped PRoPE (Projective Relative Positional Encoding) mechanism that injects per-arm SE(3) transformations and gripper states directly into attention heads; a lightweight auxiliary depth branch to enforce scene-level 3D geometry; and object-centric supervision using SAM3 masks alongside a frozen V-JEPA teacher to maintain object consistency during grasping and manipulation. Additionally, a distribution-matching distillation (DMD) method is applied to convert the multi-step generative model into a fast, few-step sampler for efficient deployment.
DreamX-Phi 1.0 successfully grounds video predictions in bimanual robot control while preserving rigid-body motion and fine-grained object interactions. On the WorldArena 2.0 Challenge, the model achieves first place on Track 1 and second place on Track 2. Furthermore, it attains an offline EWMScore-P of 76.88 on WorldArena 1.0 Track 1, demonstrating that combining geometry-aware action conditioning with specialized supervision for depth and object state yields superior physical fidelity compared to generic action token injection.
Video world models hold significant promise for robotic planning and evaluation, but existing generators frequently produce visually convincing videos that violate physical laws, miss targets, or move the wrong arm. By tightly coupling robot kinematics, depth, and object tracking into a video diffusion transformer, this work bridges the gap between general video generation and precise, control-faithful robotic simulation.
We present \textbf{DreamX-Phi 1.0}, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm's commanded path, we inject per-arm $\mathrm{SE}(3)$ transformations into attention via \textbf{PRoPE-style geometric encoding}, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweight \textbf{depth branch} for scene-level geometry and use \textbf{SAM3 masks} with a frozen \textbf{V-JEPA teacher} to maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student via distribution-matching distillation for efficient deployment. At the time of writing, \model{} achieves first place on Track~1 and second place on Track~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.
Sam: That covers the arm itself. But what about the object being picked up — the glass, say? Does the model understand how that should behave?
Alex: That's the second layer of the system. Knowing where the arm is doesn't automatically tell the model what happens to the object it touches. So they add what the paper calls extra supervision. One component tracks the depth of the scene — essentially, how far away every object is from the camera at each moment. Another is a separate "teacher" model that watches the generated video and checks whether objects remain consistent and physically plausible throughout the grasp. If the glass flickers or teleports, the teacher flags it.
Sam: So it's a bit like having a physics teacher looking over the model's shoulder while it works.
Alex: That's a fair way to put it. And the combination — the geometric constraints on the arm, the depth tracking, and the consistency checking — is what separates this approach from a standard video generator.
Sam: Does it actually hold up in practice?
Alex: The paper reports that in the WorldArena 2.0 challenge, the model achieved a top-tier ranking. What the evaluators were specifically measuring was whether the robot and the objects it interacted with stayed physically connected throughout the generated video — no ghosting, no passing through surfaces, no objects vanishing mid-grasp. That's precisely the failure mode the system was designed to prevent.
Sam: So the central insight is that you can't just train a model on raw video and hope it picks up physics. You have to build the geometry of the world directly into how the model thinks.
Alex: That's the core of it. By forcing the generative process to follow the mathematics of how solid objects actually move and rotate in space, they turn what would otherwise be a visual storytelling tool into something closer to a reliable physical simulator. Whether that's enough for robots to plan real-world actions from these imagined videos is still an open question — but it's a meaningful step toward closing the gap between looking real and being real. Thanks for listening to ResearchPod.