ENVACE: INTERNALIZING ENVIRONMENT DYNAMICS VIA WORLD REHEARSAL FOR AGENTIC REINFORCEMENT LEARNING | Zishan Xu et al. | ResearchPod

Transcript: ENVACE: INTERNALIZING ENVIRONMENT DYNAMICS VIA WORLD REHEARSAL FOR AGENTIC REINFORCEMENT LEARNING

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new approach to training AI agents called EnvACE.

Sam: So is this paper basically trying to solve the problem of how we teach AI to use tools without needing a massive, expensive simulator for every single task?

Alex: Exactly. The core challenge is that current methods rely on an external environment to tell the agent whether its actions worked. That's both costly and difficult to scale.

Sam: And the solution—EnvACE—is to have the agent simulate its own world?

Alex: That's the central idea. Instead of waiting for outside feedback, the agent learns to play two roles at once: the actor, which decides what action to take, and the environment, which generates what happens in response to that action.

Sam: So it's like a chess player who practices by playing both sides of the board—learning to predict what the opponent would do, rather than just reacting to a static puzzle?

Alex: That's a helpful way to picture it. By generating both the action and the reaction, the agent internalizes the rules of the world directly into its own memory, rather than relying on an external system to enforce them.

Sam: Is it essentially "dreaming up" its own training data, or is it more about understanding the causal link between what it does and what happens next?

Alex: It's the latter. The agent is learning the causal relationship between actions and outcomes. Researchers call this "world modeling." The key difference from older approaches is that it does this without needing a hard-coded, external simulator built specifically for each task.

Sam: But if it's doing all of this internally, how do we know it isn't just hallucinating responses that don't reflect reality?

Alex: That's a critical question. The researchers address it by tying both roles together with a single "task-success" reward. The agent only gets credit when it actually achieves the goal in a real scenario—not just when its internal simulation sounds plausible.

Sam: So the reward only comes when the real outcome works out, which forces the internal rehearsals to stay accurate?

Alex: Precisely. If the internal simulation doesn't lead to genuine success, the model gets no reward—which pushes it to keep refining its internal picture of how the world works.

Sam: And this pays off at test time too, right? Like a private rehearsal before the agent commits to a real action?

Alex: Yes. The agent can run through potential outcomes internally before taking a risk, which helps it avoid costly mistakes without needing to interact with the real environment every single time.

ENVACE: INTERNALIZING ENVIRONMENT DYNAMICS VIA WORLD REHEARSAL FOR AGENTIC REINFORCEMENT LEARNING

Zishan Xu, Zhiyuan Yao, Yuxin Chen, Yifu Guo, Zhengxi Lu, Yuquan Lu, Jinyang Huang, Yan Xu, Yasheng Wang, Weinan Zhang, Xingshan Zeng, Weiwen Liu

Aug 6, 2026·5:20·arxiv·Original Paper

World RehearsalAgentic Reinforcement LearningInternalized World ModelPrivate RehearsalEnvACEPOMDP

Listen

5 min

Briefing

Research Approach

EnvACE addresses the high cost and complexity of training LLM agents in external environments by replacing external interactions with a process called world rehearsal. Instead of querying a real environment or a separate simulator for feedback, the agent policy is trained to play two roles: an actor that generates tool calls and an environment that produces the corresponding responses. By alternating between these roles and jointly optimizing them using task-success rewards, the policy internalizes the causal relationship between its actions and the resulting environment states. This creates an integrated world model within the policy parameters, allowing the agent to anticipate outcomes and refine its decision-making process internally.

Key Findings

Experimental results across four benchmarks (BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench) demonstrate that EnvACE consistently outperforms existing environment-scaling baselines. By internalizing environment dynamics, the model achieves more transferable and robust performance across diverse tool-use scenarios. Furthermore, the study shows that this internalized world model provides significant benefits at test time; agents can perform private rehearsals to evaluate multiple candidate actions before committing to a final execution, leading to improved task success rates without requiring additional external environment interactions.

Why It Matters

This research offers a scalable alternative to traditional agent training, which is often bottlenecked by the need for complex, verifiable, or costly external environments. By shifting the burden of environment modeling from external infrastructure to the policy itself, EnvACE enables more efficient training and provides a mechanism for agents to perform 'lookahead' reasoning at inference time. This approach paves the way for developing highly capable agents that can operate effectively in environments where real-time feedback is either unavailable or prohibitively expensive.

Key Terms and Definitions

World Rehearsal — A training paradigm where the policy generates its own environment responses to simulate the consequences of its actions.
Agentic Reinforcement Learning — A framework for training LLMs to interact with tools and environments to achieve long-horizon goals through trial-and-error.
Role-Wise GRPO — An optimization technique that computes separate reward baselines for the acting and rehearsal roles to ensure balanced learning of both behaviors.
Committed Execution — The final stage of an agent's task where it performs a real-world action after having internally rehearsed and evaluated potential outcomes.
Rehearsal Memory — A collection of self-generated trajectories and feedback used by the agent to guide its decision-making during test-time execution.

Abstract

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.

ENVACE: INTERNALIZING ENVIRONMENT DYNAMICS VIA WORLD REHEARSAL FOR AGENTIC REINFORCEMENT LEARNING: Research Paper Podcast