Kuan Zhang, Yukun Chen, Zhihao Yang, Yue Su, Xiangnan Wu, Zirong Chen, Run Luo, Tao Tan, Yinhe Zheng, Yiming Li
9 min
Abstract
Understanding and tracking a dynamic world are fundamental to visual intelligence. Existing video benchmarks provide limited insight into whether models can maintain a coherent understanding of evolving world state. Building such a benchmark from real-world videos is challenging because the underlying world state is partially observable, making annotations hard to obtain. To close this gap, we introduce Video-Arena, a benchmark that turns competitive games into scalable, verifiable tests of how well models understand evolving world states from videos. Specifically, inspired by how humans understand dynamic events, we evaluate spatiotemporal world state understanding and tracking through three task categories: perceiving critical events, interpreting state updates, and tracking the resulting states. Our pipeline can generate videos and aligned reference labels from game records at scale. To keep evaluation affordable and reproducible, we curate a golden set of 180 tasks across 9 games and run evaluations under two protocols: native visual understanding and tool-assisted agentic analysis. The best-performing systems achieve F1 scores of and under the two protocols, respectively. The result highlights that native models struggle to perceive events reliably, while tool-assisted systems improve event recovery but still struggle to interpret state updates and track the resulting states consistently over time. Our project page is in https://videoarena-bench.github.io
Sam: The pipeline reconstructs events from game records. For chess, a pinned version of Stockfish evaluates the positions, and fixed bands convert its values into labels. In other environments, the records directly supply quantities such as the score or a team’s economic advantage.
Alex: So the reference can be an engine’s assessment or a recorded quantity. How do they stop models simply reading those answers off the screen?
Sam: They render videos that remove answer-revealing displays, such as scoreboards and evaluation bars, while retaining the evidence needed for the task. Each game needs its own adapter to extract events, reconstruct states, and produce the video. After that, new cases require no additional handwritten questions or model-based judging.
Alex: That makes the construction scalable. What did they actually evaluate, rather than just propose generating?
Sam: Their fixed evaluation set contains 180 cases across nine games. It includes board and card games, competitive action and strategy games, and sports. The same cases support both evaluation conditions, so the comparison does not switch to an easier collection when tools become available.
Alex: What changes between those conditions? “Using tools” can mean anything from seeing extra images to outsourcing the whole problem.
Sam: In the native visual condition, the model receives sampled frames with timestamps and produces one structured answer without tools. In the agentic condition, meaning tool-assisted, repeated analysis, it gets the full rendered video. It can extract frames, write scripts, process images, and keep working notes.
Alex: That sounds like a much stronger setup. Is it allowed to consult a game engine or retrieve the original match?
Sam: No. Source records, reference answers, external evaluators, and external model downloads are prohibited. The intended task is reconstructing the situation from the video, not finding another route to its labels. Both conditions use the same task definitions and scoring rules.
Alex: Before the results, what does success mean? Would spotting most of the action earn a respectable score?
Sam: They use strict F1, a score balancing missed events against incorrect predictions. An event earns full credit only if its identity, required entities, state update, and resulting state all match. Its timestamp must also fall within three seconds of the reference.
Alex: Then a low score does not necessarily mean the model saw nothing. It could recognize the action but misjudge its consequences.
Sam: That’s why they also diagnose the stages separately. Under native visual evaluation, the best system scores about thirteen out of a hundred. Under tool-assisted evaluation, the best reaches fifty-one. Those leaders are different systems: GPT-5.6-Sol without tools, and Claude-Opus-5.0 with Claude Code.
Alex: Could the native result mostly reflect missing evidence? Sparse frames might skip the event the model is asked to identify.
Sam: Some evidence is lost that way. But their idealized sampling ceiling is about ninety, assuming every other answer field is correct. So sampling alone does not explain the large gap. In a separate frame-budget experiment, more images improve event localization, but state judgment can stop improving.
Alex: That suggests two different failure points: finding the event and making sense of it. Do the game-specific results support that split?
Sam: They do, for the systems examined. In football and Counter-Strike, strong tool use can recover structured event records, after which score tracking works well. Board games show the opposite pattern: the strongest agent can recover every move on the diagnostic cases yet still misjudge who leads afterward.
Alex: The useful tool behavior, then, is not just opening more pictures. It’s turning those pictures into records that can be maintained and checked.
Sam: That is the authors’ interpretation. Their case analyses show scripts extracting events and building structured representations. But agentic results belong to a model and its harness together—the software connecting it to tools. They do not isolate the model’s ability from that surrounding system.
Alex: What keeps a resourceful agent from accidentally finding the answers? That seems especially important when records and evaluators exist elsewhere.
Sam: Early tests exposed exactly that risk, including ground-truth retrieval and downloaded evaluators. The authors tightened sandbox isolation and report results from the hardened setup. They describe file checks, access audits, and explicit failure categories, making protocol integrity part of the benchmark rather than an assumption.
Alex: And how far should we generalize? A running score is cleaner than most situations a real-world assistant would need to understand.
Sam: That is the main boundary. These games do not cover all dynamic environments, and real-world states cannot always be reduced to deterministic variables. Engine-derived advantage labels are also approximations, not objective truth. Their stability checks support the discrete labels, but cannot erase that distinction.
Alex: There is also a difference between a stable reference answer and a repeatable model result. Did they test repeated agent runs?
Sam: Each system-case pair has one effective run. Their uncertainty analysis measures sensitivity to which cases were selected, not randomness in the agent’s behavior. My reading is that the capability gap is well supported on this benchmark, while run-to-run reliability and transfer beyond games remain less settled.
Alex: Who should read the full document, and where should they start?
Sam: Researchers evaluating video models or building tool-using agents should read it. Start with Task Definition and Notation, then Evaluation Protocols and Metrics. The section Why models fail: tier-wise diagnosis explains the central result. If you plan to reuse the benchmark, read the sandbox appendix before treating scores as comparable.
Alex: And for someone who only needs the lesson, rather than the implementation?
Sam: Recovering what happened is not the same as understanding how the situation changed.
Alex: Thanks for listening.