Anton Hristov, Aleksandar Slavov, Vasil Vasilev
8 min
Abstract
Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty. We present an evaluation framework that separates what frozen representations contain, how they organize appearance and physical differences, and what changes a model's own prediction error. On IntPhys2, a benchmark of physically possible and impossible events, we compare representations from latent prediction, masked pixel reconstruction, and simple visual statistics. Appearance remains easy to decode, while V-JEPA2 exposes modest information about whether events are possible or impossible. Looking at the latent geometry, appearance changes become smaller compared with differences between videos, their directions line up more consistently across scenes, and their overlap with physical changes decreases in deeper layers. Across action-conditioned models, incorrect actions raise prediction error, while appearance changes also produce substantial responses. In Physion++ video clips, repeating a single frame can produce higher error than the original clip, while changing color throughout the clip can lower it. Using Kubric, a simulation and rendering framework, we find that impossible events can produce lower error than possible events. A fixed error threshold also incorrectly flags many possible events as impossible. A model can therefore represent physical information and respond to incorrect actions without its native prediction error reliably measuring physical surprise.
Alex: And falling reverses that relationship. It is lawful through the gap, but impossible if the sphere passes through the solid platform.
Sam: The stopping trajectories match across settings, as do the falling trajectories. Within each setting, both continuations share the same context frames and prediction. The test asks whether error follows physical validity when the meaning of stopping and falling reverses.
Alex: The support still changes what the camera sees, though. This isn’t a visually identical scene with a hidden physics switch.
Sam: Correct. Support appearance, shadows, and occlusion differ across settings. Across 256 configurations, the model gives stopping higher mean error in both settings. That means the impossible event scores lower than the lawful event on solid support.
Alex: So the score follows the trajectory ordering even when physical validity reverses. Does changing the video’s appearance remove that result?
Sam: No. The reversal holds both for the original appearance and when the authors shift hue throughout the clip. The uncertainty intervals exclude zero in opposite directions for solid support and the gap. Averaging those conditions would hide the disagreement.
Alex: Suppose I don’t care about the precise error difference. I just want a threshold that flags impossible clips. How well does that work?
Sam: They fit one threshold on separate calibration scenes and apply it unchanged. On clean evaluation clips, balanced accuracy is about 54 percent, averaging accuracy for lawful and impossible events. That modest average conceals sharply different performance across the support settings.
Alex: What would that look like in practice? An average near chance could hide either false alarms or missed violations.
Sam: Here, about 69 percent of lawful clips trigger false alarms. Every lawful stopping clip on solid support gets flagged. The authors don’t specify an application’s acceptable error rate, but the average alone clearly misses a consequential failure pattern.
Alex: Could this simply mean the model never learned any physical information? That would make the distinction less surprising.
Sam: They test that separately on IntPhys, a benchmark pairing possible events with physical violations. A linear probe—a simple classifier trained on frozen features—can extract physical labels from the main model. It reaches about 58 percent accuracy against chance at 50 percent.
Alex: That is modest evidence, not broad physical reasoning. Did they keep scenes separate between training and evaluation?
Sam: Yes, using scene-level cross-fitting across 57 held-out scenes. The model outperforms both a masked-pixel reconstruction model and simple visual statistics on the task combining appearances. But this readout analysis is post-hoc, and its uncertainty does not include refitting the classifiers.
Alex: Appearance remains available too, then. Does that mean the physical information and appearance information are hopelessly mixed together?
Sam: The results don’t establish that. Appearance classification reaches about 98 percent, though that task and physical classification don’t measure information on a common scale. Across network depth, appearance changes become smaller relative to differences between unrelated videos, and overlap less with physical changes.
Alex: That sounds like useful structure developing inside the model. Why doesn’t it justify interpreting its error as physical surprise?
Sam: Because feature structure doesn’t tell you what drives the predictor’s discrepancy. The authors test that connection with a separate regression. Their selected geometric measurements add little predictive value beyond scene conditions and simple visual statistics, under the tested linear models.
Alex: What about models supplied with actions? If incorrect actions raise error for the same observed outcome, isn’t that stronger evidence?
Sam: It establishes action sensitivity, which the study finds across the tested action-conditioned models. But color changes also raise error while the simulated physical state stays fixed. Using the action does not make the score specific to physical violations.
Alex: Comparing color with action sounds vulnerable to different starting difficulties. A moving target might already be harder than a static one.
Sam: The paper explicitly separates those baselines. For the models using a shared visual encoder, absolute scores and increases over each intervention’s own baseline give opposite orderings. Neither comparison makes color and action changes equally predictable or equally relevant to the task.
Alex: Do the other videos show the same kind of dependence, beyond these rendered action tasks?
Sam: In Physion plus plus, videos of physical interactions, repeating a single frame can produce higher error than the original moving clip. Persistent color shifts and darkening can lower mean error. But this implementation encodes the whole window before selecting context, so target frames can influence the predictor.
Alex: That makes those diagnostics evidence about this particular scoring implementation, not clean tests of forecasting from past frames alone. Where does that leave the overall claim?
Sam: Strongest in the controlled support experiment, and narrowest there too. It samples one procedural family with restricted cameras, colors, and lighting. Real-world and material generalization are untested, and the results establish neither absent physical knowledge nor a general limit on planning.
Alex: Who should read the full document, and where should they start?
Sam: Anyone using world-model error as physical surprise should start with the evaluation framework, then the physical-violation ordering and threshold-transfer results. Check whether your own evaluation separates trajectory, appearance, and validity. For everyone else: a model can know something about physics without its errors telling you when physics is broken.
Alex: Keep that distinction close when you interpret the next score.