Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan Liu, Xin Lai, Chenxin Li, Yiduo Guo, Zhexin Zhang, Xinyuan Wang, Tianyi Bai, Ziniu Li, Benyou Wang
4 min
Abstract
Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.
Alex: So the whole evaluation chain is automated. How are the agents actually doing against this standard?
Sam: Not well. Even the most capable agent they tested scored only around 41%. Most scored well below that. The gap between "writing code" and "building a playable game" turns out to be substantial.
Alex: Why are they failing so consistently?
Sam: The authors find that agents often produce fragments of a game. Maybe one mechanic works—a character moves, or a button responds—but the pieces don't connect into a complete, playable experience. They struggle with what you might call the "glue": things like consistent game rules, visual feedback that tells the player what's happening, or a win condition that actually triggers.
Alex: Is there anything that separates the agents that do better from those that don't?
Sam: Yes, and it comes down to how they "see" their own work. Agents that look at their own rendered screen—actually inspecting the visual output as they build—perform meaningfully better. It turns the process from a one-shot code dump into a cycle of "look, check, and fix." The ones that skip that step stay blind to visual bugs that would be obvious to any human player.
Alex: It's like the difference between a painter who steps back to look at the canvas and one who just stares at the tip of the brush.
Sam: That's a good way to put it. The paper suggests that success in this domain isn't just about generating more code—it's about closing the feedback loop between what you've built and what you can observe.
Alex: So the core finding is this: AI agents are reasonably good at writing code, but building a game requires something more. It requires navigating a real environment, assembling a complete set of parts, and then actually checking whether the result works the way a player would experience it.
Sam: That's the argument. And the benchmark is designed specifically to measure that gap—not to ask whether an AI can write code, but whether it can build something a person can actually pick up and play.
Alex: It's a meaningful distinction, and one that has real consequences for how we think about what AI can and can't yet do. Thanks for listening to ResearchPod.