Tongsheng Ding, Zhen Luo, Yixuan Yang, Boyu Wang, Luyang Xie, Jinyu Yang, Feng Zheng
5 min
Abstract
Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.
Alex: That's a meaningful gain in speed. But how does it know what kind of motion to predict in the first place?
Sam: It's guided by a plain-language instruction—something like "move the mug to the coaster." The system reads that instruction and uses it to focus its attention on the right kind of motion. The researchers also trained it on a new dataset they call "Move," which contains thousands of carefully checked examples of object motions, each paired with a detailed description of what's happening.
Alex: Why did they need a new dataset? Couldn't they use existing ones?
Sam: Most existing datasets use very coarse labels—things like "pick up" or "put down." That tells a model almost nothing about the actual path an object takes. The Move dataset provides much finer descriptions, capturing the specific character of each motion. That richer guidance is what allows the model to make precise predictions rather than vague approximations.
Alex: But if the training mixes real videos with generated predictions, doesn't that create a problem? The generated data isn't perfectly accurate.
Sam: That's a genuine challenge, and the researchers address it directly. They attach a label to each training example—essentially a tag that tells the system whether it's looking at a real recording or a generated one. That way, the model learns to treat the two sources differently, using real recordings as its anchor for what's true while still benefiting from the much larger pool of generated examples.
Alex: So it learns to be appropriately skeptical of its own generated data. What about the broader limitations? If the underlying model is fixed and not being updated, can it handle situations it's never encountered?
Sam: That's the central trade-off. DreamTraj is reading the motion knowledge that's already built into the frozen model. If the model was never exposed to a particular type of motion during its original training, there's nothing for DreamTraj to read out. It's a powerful tool for interpreting what the model already understands—but it isn't learning new physical principles on the fly.
Alex: So the ceiling on what it can do is set by whatever the underlying model already knows.
Sam: Precisely. The paper is transparent about this. The approach is efficient and practical, but its reach is bounded by the knowledge already inside the backbone. The researchers suggest that finding ways to expand that knowledge base without sacrificing speed is the natural next step.
Alex: It's an interesting design choice. You trade some flexibility for a lot of speed and simplicity.
Sam: That's the core argument of the paper. By focusing on the model's internal signals rather than its finished output, DreamTraj becomes both faster and more practical for real-world robotics—while being honest about where the boundaries of that approach lie. Thanks for listening to ResearchPod.