AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments

As language models transition from conversational chatbots to autonomous agents, there is a critical need to evaluate their performance in realistic, production-like environments. Most existing benchmarks rely on idealized, closed-world settings with pre-packaged tools and clean inputs. The authors introduce AgentGym2, an evaluation framework designed to bridge this gap by grounding tasks in real-world demands. The framework evaluates agents across 27 domains, focusing on their ability to perform end-to-end procedures, discover tools through proactive exploration, compose novel workflows, and maintain robustness against noisy or underspecified information.

Experiments on 15 proprietary and open-source models demonstrate a significant performance gap between current state-of-the-art systems and the requirements of real-world deployment. Even top-tier models like GPT-5 and Claude 4.5 Sonnet fail to achieve satisfactory success rates, often struggling with data wrangling and complex tool discovery. The study finds that while model scaling and agentic post-training improve performance, increasing the number of interaction rounds does not yield consistent gains. Fine-grained analysis reveals that incorrect analysis and insufficient exploration are the primary failure modes, highlighting that agents often lack the ability to sustain multi-turn decision processes or effectively utilize environmental feedback.

AgentGym2 provides a more rigorous and realistic testbed for AI agents, exposing the limitations of current architectures in handling the ambiguity and noise inherent in real-world workflows. By emphasizing end-to-end task completion and dynamic tool acquisition, the benchmark serves as a guide for future research into building more robust, reliable, and productive agents that can operate effectively outside of sandboxed, idealized environments.