Video generation models have made impressive strides in synthesizing visually compelling content, yet their outputs remain confined to the virtual domain. A natural question follows: how well do these models reflect the physical world when their generated videos leave the screen and enter reality? We propose robotic manipulation as a concrete, measurable window onto this question: if a model has truly internalized physical laws, the motion it depicts should translate into executable robot behavior. We introduce Dream$.$exe, an evaluation framework that operationalizes this criterion through a video-to-execution pipeline. Given a scene image and a task description, Dream$.$exe synthesizes a manipulation video, converts the generated motion into robot trajectories, and executes them in a physics simulator, yielding a grounding signal that purely visual metrics cannot offer. Using this pipeline, we evaluate 8 models spanning frontier closed-source generators, open-source generators, and robot-specific models. Our benchmark covers 101 manually curated manipulation tasks at three levels of physical complexity, measured across visual quality, trajectory fidelity, and execution success. Encouragingly, several models achieve measurable execution success, suggesting that generative priors learned from internet-scale data already encode meaningful physical knowledge. Yet visual quality proves a poor predictor of executability, exposing a dimension of model capability that standard visual evaluations do not capture. Dream$.$exe will be open-sourced at https://github.com/showlab/Dream.exe.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a study that asks a deceptively simple question about AI-generated videos: do they actually understand physics, or are they just very convincing fakes?
Sam: The framework at the centre of this study is called Dream.exe. And the puzzle it's trying to solve is this — when an AI generates a video of a robot picking up a cup or opening a drawer, has it genuinely learned how the physical world works? Or has it just learned to produce images that look right to a human eye?
Alex: That's a meaningful distinction. We've been judging these models by how polished the videos look, rather than whether the motion is actually possible.
Sam: Exactly. And the assumption baked into a lot of AI research is that if a model produces realistic-looking video, it must have built some internal understanding of the world — what researchers call a "world model." But this study challenges that directly. It suggests visual quality is a poor predictor of whether a robot could actually carry out what the video is showing.
Alex: So how do you even test that? You can't just watch the video and guess.
Sam: Right. The researchers built what they call a "video-to-execution pipeline." The idea is to take a 2D video — the kind an AI generates — and convert it into actual 3D movement instructions for a robot. They track the position of the hand across frames, estimate how far away objects are, and build a rough 3D map of the scene.
Alex: And then they hand those instructions to a real robot?
Sam: Not quite — they hand them to a physics simulation. It's a piece of software called MuJoCo, and think of it as a digital world where gravity is real, surfaces are solid, and objects have weight. If the robot's planned movement tells it to reach through a table, the simulation immediately flags that as impossible. The physical rules catch what the human eye missed.
Alex: So instead of asking "does this video look real," you're asking "could a robot actually do this?"
Sam: Precisely. And that's a much harder test to pass. They ran this across 101 different tasks — ranging from simple reaches to more complex multi-step actions — specifically to map out where a model's grasp of physics starts to break down.
Alex: Here's what I don't quite understand, though. These models are trained on enormous amounts of internet video. Shouldn't they just absorb how physics works from all of that footage?
Sam: That's the intuitive assumption, but there's a real problem with it. Internet video is just pixels — patterns of light and colour changing over time. The video doesn't contain the actual forces involved. A model can watch a thousand videos of someone lifting a heavy box, but it never feels the weight, never experiences the resistance. It's a bit like watching hours of footage of someone swimming and then assuming you know how to swim. The visual pattern is there, but the physical understanding isn't.
Alex: So the model learns what things look like when physics happens, but not why they happen that way.
Sam: That's a good way to put it. And that gap shows up clearly in the failure modes the study identifies. The most common ones are: objects appearing to float slightly above surfaces, the robot's hand passing through a solid object rather than stopping at it, and joints bending in ways that would be physically impossible for a real arm. Each of these looks fine in a video — your eye glosses over it — but a physics engine catches it immediately.
Alex: Which means we've been using the wrong measuring stick entirely.
Sam: That's the study's central argument. They found no meaningful correlation between how realistic a video looks to a human rater and how well the robot actually performs when those movements are executed. A video can score highly on visual quality and still fail completely in the simulation.
Alex: So what's the alternative? How should we be measuring these models?
Sam: The study proposes what they call a three-track evaluation. Rather than just asking "does this look good," you separately assess: how visually fluent is the video, how physically grounded are the movements, and how well does the robot actually execute the task. Splitting those three things apart forces you to be honest about what a model has and hasn't learned.
Alex: Does that mean current models aren't ready for real robotics applications?
Sam: The study's position is cautious but not dismissive. These models aren't ready for autonomous physical control — that's clear. But they do represent a meaningful step toward something the researchers call a "behavioural prior." The idea is that if you could eventually teach a robot the rough shape of a task from video — the general sequence of movements, the spatial relationships involved — you might reduce the amount of expensive, time-consuming physical training needed. Video becomes a starting point, not a finished answer.
Alex: So the dream and the action are still separate things, but the gap might be closeable.
Sam: That's where the research points. The name Dream.exe is deliberate — it's asking what it would take to turn a dreamed-up video into something a machine can actually execute. Right now, the physics aren't there. But having a clear, testable framework for measuring exactly where the gap is — that's how you start closing it.
Alex: It's a useful reminder that looking right and being right are two very different things — especially when you're handing control to a machine.
Sam: And that's true well beyond robotics. Whenever we use AI to plan or predict physical actions, we need to be asking not just "does this seem plausible" but "would this actually work." This study gives us a more rigorous way to ask that question.
Alex: Thanks for listening to ResearchPod.