The shift from video generation to interactive world modeling places new demands on data: beyond captioned videos, world models require temporally aligned video-action-language trajectories grounded in the actions, camera motion, states, and events that drive future scene changes. However, such data is difficult to obtain at scale. Web video datasets offer broad visual coverage but lack executable actions and reliable states; robotic datasets provide action and state supervision but are costly and limited in scene diversity; and existing simulators often lack large-scale human-driven interaction trajectories. In this paper, we introduce EgoCS-400K, a large-scale replay-grounded egocentric Counter-Strike dataset for world models, built from public professional CS and CS2 match demos that preserve human gameplay trajectories and enable parsing, replaying, rendering, and temporal alignment. We extract player states, view directions, movements, keyboard/button inputs, view-angle changes, weapon usage, game events, and round-level context, and render clean first-person videos from the same trajectories. EgoCS-400K contains over 400,000 first-person videos and 10,000 hours of gameplay from more than 1,000 matches and 40,000 rounds, covering 13 maps and 10 player viewpoints per round. It supports a range of interactive visual modeling tasks, including action-conditioned future prediction, state- and event-aware scene rollout, replay-grounded captioning, and agent egocentric action understanding. By connecting visual observations with human actions, camera motion, game states, and events at scale, EgoCS-400K serves as a practical bridge between passive web videos, controllable game simulation, and costly real-world embodied data.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new way to train AI to understand the world — and it involves something you might not expect: video game replay files.
Sam: That's right. The paper introduces a dataset called EgoCS-400K. The core puzzle it's trying to solve is this: how do you teach an AI that actions cause visual changes — that if you press a button, something in the world shifts — without spending millions of dollars collecting real-world data?
Alex: So the problem isn't just getting more video. It's getting the *right kind* of video — where you know exactly what caused what?
Sam: Exactly. Right now, researchers have two options, and both have serious drawbacks. On one end, there's a huge ocean of video on the internet — people cooking, walking, playing sports. But watching a video doesn't tell you *why* someone moved their hand. You're just seeing the result, not the cause. On the other end, you have robotic data — where a robot actually performs a task and logs every movement. That's ideal, but collecting it at scale is extremely expensive and slow.
Alex: So there's a gap between "lots of data" and "useful data."
Sam: Right. And the researchers argue that video games sit right in the middle of that gap in a useful way. Here's why: when a professional player competes in a game like Counter-Strike, the game doesn't just record video. It saves a replay file — essentially a flight recorder for the entire match. Every button press, every mouse movement, every change in the game's internal state is logged with precise timing.
Alex: So it's not like watching a recording of the match on YouTube. It's more like having the raw data tape from the cockpit?
Sam: That's a good way to put it. Because the game engine is a consistent, rule-based system, researchers can take that replay file and *re-run* it. When they do, they can extract perfectly synchronized video frames, the exact player inputs at each moment, and the game's internal state — all lined up together. No guessing, no manual labeling. The game itself provides the ground truth.
Alex: And that's where the scale comes from. They turned thousands of these replay files into a usable dataset?
Sam: Yes. The paper reports converting over ten thousand hours of professional gameplay into around four hundred thousand video clips, each one tagged with the actions that caused it.
Alex: That's a meaningful amount of data. But I imagine just having a lot of clips isn't enough. How do they make sure the AI is actually learning something useful, rather than just memorizing patterns from one specific game?
Sam: That's the right question to ask. The researchers address it with what they call "multi-grained annotation." Think of it like learning basketball. A coach doesn't just show you a full game and say "figure it out." They break it down — footwork, passing angles, defensive positioning — and then show you how those pieces connect to a play. The dataset does something similar. It doesn't just label a clip as "gameplay." It generates structured descriptions at multiple levels of detail: what the player's hands are doing, what the immediate goal of that action is, and how it fits into the broader flow of the match.
Alex: So the AI is seeing the same moment described from different angles of zoom, so to speak?
Sam: Precisely. And to generate those descriptions automatically, they use a tool called a Vision-Language Model — think of it as a system that can look at an image and write a sentence explaining what's happening in it, the way a caption writer would. Because the replay file tells them exactly what the player input was at every moment, the captions can be grounded in fact rather than guesswork.
Alex: You mentioned the timing is very precise. How precise are we talking?
Sam: The game engine operates in units called "ticks" — you can think of each tick as one heartbeat of the simulation. It's finer-grained than a video frame. By aligning every visual frame to a specific tick, the system knows not just *what* the player did, but *when* — down to a fraction of a second. That level of precision is what makes the dataset different from just recording a screen.
Alex: And I imagine cutting all that footage into clips isn't straightforward either. You can't just chop it up randomly.
Sam: Right. The researchers use a technique from computer science — a method for finding the most efficient solution to a problem by breaking it into smaller steps — to figure out where to cut the footage. The key constraint is what they call "protected action chains." Some actions in a game are one continuous logical unit. Reloading a weapon, for example, has a beginning, a middle, and an end. If you cut in the middle of that sequence, the AI learns a broken, incomplete behavior. So the algorithm is specifically designed to keep those sequences intact when it decides where to split clips.
Alex: So the structure of the dataset reflects the structure of human intention, not just the structure of time.
Sam: That's a clean way to put it. The goal is that when the AI trains on this data, it learns that actions have consequences — that a specific input leads to a specific visual change — and that those changes unfold over time in a logical sequence. That's the kind of understanding you need if you ever want an AI to act in the real world, not just recognize what it sees.
Alex: Which raises the obvious question. Games are simulations. Does any of this actually transfer to the real world?
Sam: The paper is candid about this. Games are not the real world. The visual style is different, the physics are simplified, and the range of actions is constrained by what the game allows. The researchers frame EgoCS-400K as a step toward building better world models — AI systems that understand cause and effect — not as a finished solution. The argument is that learning the *structure* of action and consequence in a controlled environment is a meaningful foundation, even if the specifics don't transfer directly.
Alex: So it's less "the AI learned to play Counter-Strike" and more "the AI practiced understanding that actions have consequences" — and that practice might carry over.
Sam: That's the hypothesis. And the value of the approach is really in the method: using existing game replay infrastructure to generate structured, labeled, action-grounded data at a scale that would be impractical to collect any other way. Whether that transfers, and how well, is still an open question — but the dataset gives researchers a tool to start investigating it seriously.
Alex: That's a genuinely interesting framing. Instead of building expensive robots or hiring armies of human labelers, you mine the data that competitive gamers have already been generating for years — and you do it in a way that preserves the causal structure that makes it useful.
Sam: And that's what makes the approach worth paying attention to. The hard part of teaching AI to act in the world isn't just showing it enough video. It's showing it video where the *why* is legible. Game replay files, it turns out, are one of the few places where that information already exists, at scale, for free.
Alex: Thanks for listening to ResearchPod.