Kaiwei Chen, Moran Li, Zhenye Gan, Yabiao Wang, Chengjie Wang, Kexin Zhang, Yong Liu, Jiangning Zhang
9 min
Abstract
Recent video generation models produce visually compelling outputs, yet their physical plausibility remains poorly measured. Physical plausibility asks whether generated objects obey geometry, kinematics, dynamics, and conservation laws. Existing benchmarks rely on coarse visual-language model judgments or target embodied world models rather than video generation itself. We introduce PVBench, a physical video benchmark that quantitatively measures the plausibility of generated videos across 46 dimensions organized into seven domains, spanning: i) instruction and semantic fidelity, ii) behavioral and causal consistency, iii) perceptual and temporal quality, iv) geometric and spatial consistency, v) material properties and field states, vi) kinematic and spatiotemporal persistence, vii) and mechanical plausibility with conservation laws. PVBench comprises 600 carefully crafted prompts, of which 30% involve physical embodied agents such as robot arms, humanoids, quadrupeds, and multi-agent systems. For challenging physically tractable dimensions, it pairs each with specialized vision models that derive verifiable, physically grounded measurements, such as object-level 6D pose, metric scale, surface normals, kinematics, and numerical conservation-law residuals. It thus avoids relying solely on subjective scoring, while supporting both text-to-video and image-to-video generation. Finally, we validate the agreement between automatic metrics and human judgments through human evaluation. Meanwhile, we evaluate mainstream open-source and closed-source models and present several insights. By releasing the benchmark, prompts, and calibration protocol, we aim to foster physically grounded video generation.
Alex: The prompt specifies the objects' mass ratio. Specialized vision models identify the objects and track their motion before and after impact. The evaluator combines those trajectories with the mass ratio to estimate how much total momentum changes.
Sam: So the question becomes whether the motion and stated masses fit momentum conservation, rather than whether the collision looks familiar.
Alex: And the size of that mismatch feeds into the score. Other dimensions use different measurements. For depth, the evaluator checks whether an occluded object is estimated to sit behind its occluder. For shape, it compares reconstructed object geometry across frames.
Sam: But material behavior or a phase change sounds much harder to recover numerically. Does everything really become a physical measurement?
Alex: No. Most physical dimensions receive numerical evaluation, but some combine measurement with language-model judgment. Others rely entirely on language-model scoring, including material properties and phase transitions. The benchmark is measurement-led, not measurement-only.
Sam: The prompts must matter here. If the relevant objects disappear behind a wall, even a good evaluator cannot measure the collision.
Alex: The authors built 600 prompt groups, each pairing text-to-video with image-to-video versions of a scene. Text-to-video starts from a written description. Image-to-video also receives a reference image establishing the initial scene. Prompts are screened for physical plausibility, visual feasibility, and whether the target quantities are observable.
Sam: That pairing lets them ask whether supplying the starting appearance also improves later physics. How broadly did they test it?
Alex: They evaluated six video-generation models in both modes, using a common output specification and fixed random seed. The set includes open and closed systems. Each model was asked to generate the full prompt collection.
Sam: What pattern matters more than the leaderboard?
Alex: Geometry and general motion scored better than material states and mechanical conservation. Mechanics stayed weak across the evaluated models. A continuous trajectory or a completed action did not establish that forces, collisions, or energy changes were physically reasonable.
Sam: And did a starting image help? It could constrain the scene without constraining what happens afterward.
Alex: The results support that distinction, but not a universal effect. In their detailed MiniMax comparison, image conditioning improved instruction following and perceptual quality. Physical results were mixed: material-state scores improved, while geometry and mechanical conservation declined. The paired comparisons differ across models.
Sam: So neither sharper appearance nor a supplied starting frame is a shortcut to reliable dynamics. What happened when scenes became harder to observe?
Alex: Most models scored worse on a supplemental subset involving embodied agents, occlusion, visual clutter, or camera movement. There were exceptions. That result also raises an evaluation question: harder scenes burden the generator, but they can burden tracking and reconstruction too.
Sam: Which brings us to trust. How do we know these scores measure errors that people recognize, rather than quirks of the vision tools?
Alex: The human validation included 375 video-and-dimension samples from a subset of the tested generators. Trained annotators independently scored each sample, and their ratings were averaged. Overall, automatic scores had a Spearman correlation of about point seven seven with human scores.
Sam: Spearman measures agreement in ranking. That's useful evidence that the evaluator orders examples similarly to people, but it doesn't make every recovered physical quantity correct.
Alex: The authors also directly compared numerical scoring with language-model scoring on the same physical dimensions. Numerical scores correlated with human ratings at point eight four. The corresponding language-model correlation was point one four. Language-model scores also frequently hit the maximum, making differences harder to distinguish.
Sam: That's the strongest case for the measurement approach. But an overall correlation could still hide dimensions where the evaluator performs poorly.
Alex: It does. Object permanence—whether an object maintains its identity through disappearance and return—had a correlation of only about point one three. Motion continuity was also weakly aligned. The appendix links some weaker results to tracking, detection, and unobservable or skipped samples.
Sam: Then the main limitation isn't just that generated videos break physics. It's that the tools judging them can lose the objects too.
Alex: That's a central limitation. The benchmark depends on specialized vision models, and the authors have not fully tested how replacing those evaluators changes the results. Most dimensions also lack ground-truth references. They mainly check consistency with physical laws, rather than compare against a known physical trajectory.
Sam: So I'd treat the scores as evidence about particular physical behaviors, with uneven reliability across dimensions—not as certification that a video is physically correct.
Alex: That's my reading too. Aggregate human agreement supports the benchmark's usefulness, while the per-dimension results set limits on confidence. The authors propose simulation-based references for future evaluation. Those references are a plan, not evidence supplied here.
Sam: Who should spend time with the full document, and where should they begin?
Alex: Researchers building video generators, evaluation systems, or video-based tools for embodied intelligence should read it. Start with the appendix on physical quantities and scoring principles, then the per-dimension human-alignment results. Those sections show both what each score means and where confidence weakens.
Sam: And for someone outside those areas, what's the line to remember?
Alex: A video looking right is not evidence that its physics is right. PVBench's useful move is to ask which quantities were measured, rather than accept visual plausibility as the answer.
Sam: Keep that distinction in mind when you watch the next convincing clip.