Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at why modern AI agents struggle to build actual video games, even when they're quite good at writing standard computer code.
Sam: That's the central puzzle of a new paper called "GameCraft-Bench." The researchers argue that building a game isn't just about writing lines of code. It's about creating an interactive experience that has to work inside a complex software environment.
Alex: So this paper is basically asking why current AI, which seems so capable at coding, hits a wall when it tries to act like a full-stack game developer?
Sam: Exactly. Most AI coding tools treat a game like a static document—a list of instructions to be written out. But a game is a living system that must be launched, rendered, and played. If the AI writes code that doesn't actually run inside a game engine, it hasn't really built a game. It's produced something closer to a blueprint.
Alex: So the problem isn't the code itself, but the fact that the code has to live inside a specific, demanding environment?
Sam: Precisely. The researchers identified three requirements for a successful game. The first is what they call "Engine Grounding." The AI must develop the game within a real game engine—in this case, one called Godot—using its specific tools and rules. Think of it like the difference between building a car in an actual workshop versus sketching one on a napkin. The napkin sketch might be accurate, but it doesn't go anywhere.
Alex: That makes sense. What are the other two requirements?
Sam: The second is "Artifact Completeness." The agent must deliver a project that is actually launchable, with all the necessary files and assets included—not just the main code, but everything the engine needs to start the game. The third is "Interactive Verification." You can't just read the code to know if a game works. You have to play it.
Alex: How do they actually test that? If you have to play the game to verify it, does a human have to sit there and test every single AI-generated project?
Sam: That would be impossible at scale. Instead, they use something called "replay traces." Think of it like a driving test for software. The system records a precise sequence of mouse and keyboard inputs—things like "press jump here" or "move left at this moment"—and then replays those exact inputs on whatever game the AI generated.
Alex: Oh—so that's how they turn open-ended, unpredictable gameplay into something a computer can actually grade.
Sam: Exactly. And then a "multimodal judge" takes over. That's an AI system that watches the resulting video of the gameplay and checks it against a hidden rubric—evaluating things like whether the character actually jumped when the button was pressed, or whether the visuals look coherent.
Alex: So the whole evaluation chain is automated. How are the agents actually doing against this standard?
Sam: Not well. Even the most capable agent they tested scored only around 41%. Most scored well below that. The gap between "writing code" and "building a playable game" turns out to be substantial.
Alex: Why are they failing so consistently?
Sam: The authors find that agents often produce fragments of a game. Maybe one mechanic works—a character moves, or a button responds—but the pieces don't connect into a complete, playable experience. They struggle with what you might call the "glue": things like consistent game rules, visual feedback that tells the player what's happening, or a win condition that actually triggers.
Alex: Is there anything that separates the agents that do better from those that don't?
Sam: Yes, and it comes down to how they "see" their own work. Agents that look at their own rendered screen—actually inspecting the visual output as they build—perform meaningfully better. It turns the process from a one-shot code dump into a cycle of "look, check, and fix." The ones that skip that step stay blind to visual bugs that would be obvious to any human player.
Alex: It's like the difference between a painter who steps back to look at the canvas and one who just stares at the tip of the brush.
Sam: That's a good way to put it. The paper suggests that success in this domain isn't just about generating more code—it's about closing the feedback loop between what you've built and what you can observe.
Alex: So the core finding is this: AI agents are reasonably good at writing code, but building a game requires something more. It requires navigating a real environment, assembling a complete set of parts, and then actually checking whether the result works the way a player would experience it.
Sam: That's the argument. And the benchmark is designed specifically to measure that gap—not to ask whether an AI can write code, but whether it can build something a person can actually pick up and play.
Alex: It's a meaningful distinction, and one that has real consequences for how we think about what AI can and can't yet do. Thanks for listening to ResearchPod.