Evaluating whether Multimodal Large Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing benchmarks rely on recognition-style protocols such as Visual Question Answering (VQA) and Violation of Expectation (VoE), which can often be answered without committing to an explicit, testable physical hypothesis. We propose VisPhyWorld, an execution-based framework that evaluates physical reasoning by requiring models to generate executable simulator code from visual observations. By producing runnable code, the inferred world representation is directly inspectable, editable, and falsifiable. This separates physical reasoning from rendering. Building on this framework, we introduce VisPhyBench, comprising 209 evaluation scenes derived from 108 physical templates and a systematic protocol that evaluates how well models reconstruct appearance and reproduce physically plausible motion. Our pipeline produces valid reconstructed videos in 97.7% on the benchmark. Experiments show that while state-of-the-art MLLMs achieve strong semantic scene understanding, they struggle to accurately infer physical parameters and to simulate consistent physical dynamics.
Alex: Welcome to another episode of ResearchPod.
Sam: Today, we're looking at how smart AI systems handle understanding the physical world—like how balls roll or blocks fall. These systems are called multimodal large language models, or MLLMs for short. They're AI trained on huge amounts of pictures, videos, and text, so they can describe what's in an image or answer questions about it, much like a smart assistant that sees and talks.
Alex: They seem pretty good at saying what they see. But do they really get the physics behind it, like why things move a certain way?
Sam: That's the key question. Most tests today show the AI a video and ask it to pick the right next frame or answer a question. These often just spot patterns or flag weird events. They don't force the AI to build a full model of how objects interact under real physical rules. It can guess from what it's seen before without truly understanding causes.
Alex: So it's like the AI aces describing a soccer game but might not predict the ball's bounce right if something changes. What does this research do differently?
Sam: The researchers created VisPhyWorld, a setup that pushes the AI further. It shows the AI two key frames from a video—like a ball before and after hitting blocks. Then it asks the AI to write actual computer code that recreates the whole scene and predicts what happens next using a physics simulator. That code runs in tools like Three.js, which handle real rules for gravity, bounces, and friction—just like in video games but grounded in physics.
Alex: Huh. So the code has to spell out masses, speeds, and collisions explicitly—or it fails when run.
Sam: Exactly. This makes the AI's thinking inspectable. You read the code to see what physics it inferred, edit it if needed, and check if the simulated video matches the real one. Their benchmark, VisPhyBench, tests this on 209 simple scenes. It shows these AIs nail describing scenes but often mess up the motion, like objects passing through each other instead of bouncing properly.
Alex: It reveals they're relying on looks, not real physical logic. But how do they check if the AI's code actually captures the physics right, beyond just looking similar?
Sam: They run the code through a physics engine and compare the new video it makes to the original. One check looks at how closely the pictures match by comparing small patches of color and shape—like checking if two drawings line up in outlines and shades. Another tracks how pixels move between frames, measuring if paths match—like a ball's arc curving the same way.
Alex: Okay, so multiple ways to score looks and movement. Do different code tools change how well this works?
Sam: Yes, they tested engines like Three.js and P5.js, which simulate physics such as gravity pulling objects down or balls bouncing with realistic pushback. Three.js works better because it handles rigid objects—things that don't bend—like friction and collisions without objects clipping through each other. Switching to Three.js cut visual errors notably in one top model.
Alex: Huh, so the tool matters because some bake in better physics. And how do these code setups stack up against regular video generators?
Sam: The code-driven ones, especially with Three.js, match or beat pixel-based baselines on most checks. They're strong on scene looks, but the real win is inspectability—you see exactly what physics the AI guessed, like object masses or speeds. Baselines look good sometimes but hide why motions fail, with no editable code. Gemini with Three.js topped overall scores.
Alex: That dissociation—strong on describing, weaker on physics—shows up clearer here. Still limited to basic setups, though.
Sam: Right, it's for simple rigid-body scenes, not real-world messiness or long actions—the paper notes that as a bound. But it opens doors to editable robot simulations you can tweak for safety.
Alex: Those editable sims could help robots avoid mistakes, like mispredicting a ball's path during a grab. Can you walk through a specific failure, like how a model might botch a collision?
Sam: Take one test case: balls drop into a gray container shaped like a U at the bottom, with a red circle up top connected to a short gray rod. The model wrongly assumes a full ground plane and adds fake obstacles, like an arc of black circles that aren't there. Its code lets objects overlap without bouncing—as if solid balls phase through each other. When run in Three.js, the video shows impossible motion, failing the checks.
Alex: So the AI invents scene parts to fit its guess, messing up the physics. For a robot watching a real video of that to plan a grasp, it'd predict the ball sinks through, maybe missing the catch.
Sam: Precisely. If a robot simulates objects passing through due to bad guesses on masses or friction, it fails the grasp. Humans can inspect and edit the code—like tweaking velocity or adding real walls—then re-run for a safer prediction. This makes world models verifiable for safety-critical tasks.
Alex: That gap between seeing and simulating right highlights why inspectable code matters over black-box predictions.
Sam: Yes, it exposes semantic skill without true physical fidelity—the code commits to explicit parameters, revealing illusions when executed. The approach is limited to these simple setups, but it sets a diagnostic baseline for physical reasoning.
Alex: What if the code doesn't run first try—do they just scrap it?
Sam: No, they add a retry: if rendering fails due to code errors, they feed the error log back to the model for a second shot. This lifts success rates notably. Most fixes handle surface issues, not deep scene misunderstandings.
Alex: Huh—that shows failures are often fixable tweaks, not total physics gaps. So pulling it all together, these models describe scenes sharply but their physics simulations often reveal gaps—like objects phasing through each other—because the code forces explicit rules.
Sam: Yes, the paper suggests this code-driven approach exposes strong semantic understanding alongside physical reasoning shortfalls, as seen in consistent motion errors despite good looks. It's limited to simple synthetic rigid-body scenes in 2D-like setups, without real-world videos or complex 3D. Still, the inspectable outputs offer a path to verifiable world models for robots, where humans can tweak parameters for safer predictions.
Alex: It's a meaningful diagnostic tool, highlighting where visual smarts meet physical limits. Thanks for breaking it down so clearly, Sam.
Sam: My pleasure, Alex. This work advances how we test and refine AI for the real physical world.
Alex: Thanks for joining ResearchPod.