Jiarong Liang, Max Ku, Ka-Hei Hui, Ping Nie, Wenhu Chen
6 min
Abstract
Evaluating whether Multimodal Large Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing benchmarks rely on recognition-style protocols such as Visual Question Answering (VQA) and Violation of Expectation (VoE), which can often be answered without committing to an explicit, testable physical hypothesis. We propose VisPhyWorld, an execution-based framework that evaluates physical reasoning by requiring models to generate executable simulator code from visual observations. By producing runnable code, the inferred world representation is directly inspectable, editable, and falsifiable. This separates physical reasoning from rendering. Building on this framework, we introduce VisPhyBench, comprising 209 evaluation scenes derived from 108 physical templates and a systematic protocol that evaluates how well models reconstruct appearance and reproduce physically plausible motion. Our pipeline produces valid reconstructed videos in 97.7% on the benchmark. Experiments show that while state-of-the-art MLLMs achieve strong semantic scene understanding, they struggle to accurately infer physical parameters and to simulate consistent physical dynamics.
Sam: Yes, they tested engines like Three.js and P5.js, which simulate physics such as gravity pulling objects down or balls bouncing with realistic pushback. Three.js works better because it handles rigid objects—things that don't bend—like friction and collisions without objects clipping through each other. Switching to Three.js cut visual errors notably in one top model.
Alex: Huh, so the tool matters because some bake in better physics. And how do these code setups stack up against regular video generators?
Sam: The code-driven ones, especially with Three.js, match or beat pixel-based baselines on most checks. They're strong on scene looks, but the real win is inspectability—you see exactly what physics the AI guessed, like object masses or speeds. Baselines look good sometimes but hide why motions fail, with no editable code. Gemini with Three.js topped overall scores.
Alex: That dissociation—strong on describing, weaker on physics—shows up clearer here. Still limited to basic setups, though.
Sam: Right, it's for simple rigid-body scenes, not real-world messiness or long actions—the paper notes that as a bound. But it opens doors to editable robot simulations you can tweak for safety.
Alex: Those editable sims could help robots avoid mistakes, like mispredicting a ball's path during a grab. Can you walk through a specific failure, like how a model might botch a collision?
Sam: Take one test case: balls drop into a gray container shaped like a U at the bottom, with a red circle up top connected to a short gray rod. The model wrongly assumes a full ground plane and adds fake obstacles, like an arc of black circles that aren't there. Its code lets objects overlap without bouncing—as if solid balls phase through each other. When run in Three.js, the video shows impossible motion, failing the checks.
Alex: So the AI invents scene parts to fit its guess, messing up the physics. For a robot watching a real video of that to plan a grasp, it'd predict the ball sinks through, maybe missing the catch.
Sam: Precisely. If a robot simulates objects passing through due to bad guesses on masses or friction, it fails the grasp. Humans can inspect and edit the code—like tweaking velocity or adding real walls—then re-run for a safer prediction. This makes world models verifiable for safety-critical tasks.
Alex: That gap between seeing and simulating right highlights why inspectable code matters over black-box predictions.
Sam: Yes, it exposes semantic skill without true physical fidelity—the code commits to explicit parameters, revealing illusions when executed. The approach is limited to these simple setups, but it sets a diagnostic baseline for physical reasoning.
Alex: What if the code doesn't run first try—do they just scrap it?
Sam: No, they add a retry: if rendering fails due to code errors, they feed the error log back to the model for a second shot. This lifts success rates notably. Most fixes handle surface issues, not deep scene misunderstandings.
Alex: Huh—that shows failures are often fixable tweaks, not total physics gaps. So pulling it all together, these models describe scenes sharply but their physics simulations often reveal gaps—like objects phasing through each other—because the code forces explicit rules.
Sam: Yes, the paper suggests this code-driven approach exposes strong semantic understanding alongside physical reasoning shortfalls, as seen in consistent motion errors despite good looks. It's limited to simple synthetic rigid-body scenes in 2D-like setups, without real-world videos or complex 3D. Still, the inspectable outputs offer a path to verifiable world models for robots, where humans can tweak parameters for safer predictions.
Alex: It's a meaningful diagnostic tool, highlighting where visual smarts meet physical limits. Thanks for breaking it down so clearly, Sam.
Sam: My pleasure, Alex. This work advances how we test and refine AI for the real physical world.
Alex: Thanks for joining ResearchPod.