Xinye Li, Lingshuai Lin, Lei Wang, Liuzhou Zhang, Jialin Cui, Qingshan Li, Guanchu Wang, Qingbin Liu, Xi Chen, Jiang Bian, Wai Lam
6 min
Abstract
Action-conditioned video world models require low-latency causal generation and reliable responses to game-native controls. Although causal distillation enables one- or few-step video synthesis, extending it to interactive world models remains challenging, as discrete keyboard states and continuous mouse motion must remain aligned with temporally compressed latent chunks during causal training and autoregressive rollout. We introduce ForgeWM, a progressive framework that transforms a bidirectional action-conditioned video generator into efficient few-step world models through domain adaptation, teacher-forced causal training, causal consistency distillation, and on-policy distribution matching with a bidirectional teacher. The resulting budget-specialized students operate at steady-state denoising budgets of 1, 2, and 4 steps. ForgeWM further supports a dual-path deployment protocol combining latency-critical interaction with optional replay-time refinement, where the one-step student re-noises and refines its saved draft. On paired Minecraft trajectories, ForgeWM leads the evaluated systems in Imaging Quality, reference-aligned motion-profile agreement, action-sign accuracy, and mouse-control accuracy, while achieving the lowest reference LPIPS; the same four-stage recipe transfers to gamepad-controlled FPS gameplay. Replay-time refinement matches four-step reference quality while remaining roughly three times closer to the experienced trajectory than regeneration from noise. These results demonstrate ForgeWM's effectiveness for controllable few-step video generation.
Sam: So it's building confidence before going solo.
Alex: Right. In the third stage, it learns to anticipate cues from its immediate context, and finally it performs live — generating frames one at a time, relying entirely on its own previous output. No safety net. By the end of that progression, the model has learned to stay coherent even when it can only look backward, and it keeps the player's keyboard and mouse inputs properly aligned with what appears on screen.
Sam: That's a clever way to transfer knowledge from the easier task to the harder one. But what about visual quality? If it's running fast with fewer steps, does it look worse?
Alex: That's where a second technique comes in. After a play session ends, the system can go back and refine the recorded footage offline — meaning it's no longer under time pressure. It takes the saved video, adds a small amount of controlled noise back into it, and then runs the full denoising process again at higher quality settings.
Sam: So it's not changing what the player actually did — the choices, the camera angles — just cleaning up how it looks?
Alex: Precisely. The gameplay decisions are locked in by the recording. The offline pass just sharpens the local visual details. And according to the paper, the result matches the visual quality you'd get from a much slower four-step model, while the live session still runs at one-step speed.
Sam: That's a clean separation — speed when it matters, quality when you have time.
Alex: And the same four-stage training recipe transferred to a completely different type of game — first-person shooters controlled by gamepads rather than keyboard and mouse. No changes to the underlying model architecture were needed. The researchers just widened the control interface to handle analog sticks and buttons.
Sam: So the framework isn't locked to one game or one input style.
Alex: That's what the data suggests. It points toward a general training path for interactive video generation, not a one-off solution.
Sam: What are the limitations the authors flag?
Alex: A few worth noting. First, when the model runs for extended periods — well beyond the test window used in the paper — it gradually loses structural coherence and starts producing color artifacts. That's the error accumulation problem reasserting itself over long sessions.
Sam: So it holds together for a while, but not indefinitely.
Alex: Not yet, no. The second limitation involves motion. In the multi-game setup, the simulated camera movements tended to be somewhat more exaggerated than what the controller inputs actually called for — overshooting by a factor of roughly one and a half on average.
Sam: So the system understood which direction to turn, but sometimes turned too far?
Alex: That's a fair summary. Directional accuracy and persistence — meaning whether the motion kept going in the right direction — were both solid. But the magnitude, how far the camera actually moved, lagged behind. The authors flag this as an open problem.
Sam: So there's real progress here, but also clear room to grow.
Alex: That's the honest picture. Progressive causal training and fast generation meaningfully close the gap between generative quality and real-time speed. But long-session stability and motion precision remain areas where future work is needed. It's a meaningful step, not a finished solution.
Sam: A solid foundation, then, rather than a final answer.
Alex: Well put. Thanks for listening to ResearchPod.