Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang, Seungryong Kim
8 min
Abstract
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.
Sam: The readout is the distinction. The authors contrast DreamX with AdaWorld and VideoWorld2, which use video generation but learn action representations through separate modules. DreamX uses the pretrained video backbone for both extracting motion and supervising what gets extracted.
Alex: Make that concrete. What passes through the model when it watches a demonstration?
Sam: The video becomes a sequence of internal visual tokens, or feature units. DreamX adds trainable query tokens: small summaries that gather information from those features. They produce a motion description with a three-hundred-fold reduction in token count per latent video frame.
Alex: Compression creates a bottleneck, but it doesn't guarantee motion rather than appearance. What pushes those summaries toward the right information?
Sam: The reconstruction task. The decoder receives the initial frame and the motion summaries, but no text instruction. The initial frame supplies the scene's appearance. The pretrained generator supplies learned expectations about plausible videos, which the paper calls its generative prior.
Alex: So the summaries should provide what's missing: how the scene changes. That's an incentive to separate motion from appearance, not a mathematical guarantee.
Sam: That's the right distinction. The authors say this setup encourages separation. They train on human videos and robot demonstrations together, adding an action-prediction objective when robot action labels are available.
Alex: What does transferring that description actually look like, beyond reconstructing the video it came from?
Sam: Take the human demonstration of writing the letter N. DreamX extracts its motion description, then decodes it using the target robot's initial state. The robot reproduces the stroke structure, not merely a coarse outcome like reaching a destination.
Alex: Does it need to see where its hand goes while carrying out those strokes?
Sam: Not in that transfer experiment. Execution is open-loop, meaning there are no visual observations during the motion. The robot gets its initial proprioceptive state, its internally sensed body configuration, and the transferred motion description.
Alex: That makes the example useful for testing what the description preserves. But it doesn't demonstrate a robot correcting mistakes as it goes.
Sam: Correct. Other examples preserve a forward-then-left pushing sequence and corresponding movements of both arms. Those are qualitative demonstrations of motion structure transferring without paired human–robot training data.
Alex: Let's turn to the stronger comparisons. How do they test whether these descriptions carry useful motion information?
Sam: They use LARYBench. Its regression tests ask whether the representation supports recovering end-effector trajectories, the paths of robot hands or tools. Classification tests ask whether it distinguishes different actions, including human and robot motions.
Alex: And compared with the strongest competing action representation, how large is the difference?
Sam: Against DiLA, average normalized trajectory-prediction error falls from point three eight to point one seven. Lower is better. Average action-classification accuracy rises from about sixty-two percent to about seventy percent.
Alex: That supports useful motion encoding across the benchmark's datasets. Does the appearance-separation claim get a more direct test?
Sam: They transfer human motion into generated robot videos on the HumanAndRobot benchmark. Paired robot recordings are used only as evaluation references, not for training. DreamX beats AdaWorld and VideoWorld2 on every reported video-quality metric.
Alex: Generated videos still aren't executable control. But preserving the target robot's appearance while following human motion does address the leakage problem.
Sam: And their perturbation test adds supporting evidence. Changing the motion summaries disrupts manipulation while leaving the surrounding scene largely unchanged. Removing the summaries largely suppresses motion. Together, these tests support separation, without proving perfect independence from appearance.
Alex: For a working policy, you can't always provide a demonstration of the next movement. Can the robot predict these descriptions itself?
Sam: That's the downstream policy experiment. DreamX predicts further-compressed motion codes from the current image and instruction, alongside low-level robot actions. It doesn't generate future video during execution. Human videos can supervise code prediction even without robot action labels.
Alex: Does avoiding that video generation actually help in a real task?
Sam: In drawer closing, average subgoal success was about ninety-seven percent, versus fifty percent for Mimic-Video. That measure averages reaching and pushing completion, rather than counting only fully completed tasks. DreamX's inference was about nineteen times faster.
Alex: Those are substantial differences, but drawer closing is a narrow setting. How much weight should listeners put on the policy result?
Sam: Treat it as feasibility evidence. The study covers different drawer configurations, but still one task family. The provided main text doesn't give trial counts. And “universal” describes the intended shared representation, not demonstrated reliability across arbitrary robots, environments, or tasks.
Alex: So who should read the full document, and where should they start?
Sam: Robot-learning researchers should start with the representation-quality comparisons, then the motion-transfer method. Those carry the central contribution. If deployment efficiency is your interest, read the policy evaluation carefully, especially its success definition and restricted task scope. Video-representation researchers should examine how the queries extract information from the pretrained backbone.
Alex: And for someone outside those areas, what's the line worth carrying back to a colleague?
Sam: A video model can supply more than predicted pictures: it can supply a reusable description of motion. DreamX makes that idea concrete, with strong benchmark comparisons and more limited evidence for general robot control.
Alex: Keep that distinction in mind when you read the transfer claims.