CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?

Video world models are increasingly viewed as physical simulators, yet existing evaluation suites mostly score per-sample visual quality or compare output distributions coarsely across a whole dataset. These standard methods ignore fine-grained aleatoric uncertainty. If macroscopic random events generated by a model, such as dice rolls or coin flips, deviate from genuine statistical randomness, the model exhibits hidden miscalibration even if individual frames look plausible. CaliBench bridges this gap by evaluating generative world models directly within discrete, physically interpretable outcome spaces where the true reference distributions are analytically known in closed form.

CaliBench curates nine distinct physical scenes spanning several distribution families: binomial Galton boards (both physical and animated), Bernoulli forks, walking decisions, double pendulums, uniform dice, cards, and lotteries, as well as an asymmetric European roulette wheel. For each scene, an image-to-video model generates 32 video rollouts using identical conditioning frames and text prompts while varying only the random seed. A vision-language model extracts the discrete outcome from the final frame of each video. The protocol then decomposes model performance into two orthogonal axes:

• Scorability: The fraction of generations that produce a structurally valid, interpretable outcome without dissolving into artifacts.
• Calibration: The total variation distance between the empirical distribution of scoreable outcomes and the analytical reference distribution.

Evaluating six state-of-the-art image-to-video models across these nine scenes reveals a consistent failure mode: models tend to collapse probability mass onto a small subset of outcomes rather than faithfully matching the reference distribution. Most model-scene combinations are significantly miscalibrated, with extreme cases like Veo 3.1 collapsing entirely to a single outcome on dice. Furthermore, scorability varies widely, as some models produce ambiguously placed objects (such as a rolling roulette ball) that defy automated extraction. Overall, performance varies sharply by scene rather than by model, meaning no single architecture dominates across all environments.