OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

Overview of OmniaBench

OmniaBench is a comprehensive benchmark designed to evaluate the capabilities of general-purpose AI agents across a wide spectrum of real-world scenarios. Unlike previous benchmarks that often focus on narrow tool-use cases or specific domains, OmniaBench organizes its tasks using a hierarchical taxonomy covering 90 level-1 and 354 level-2 domains, spanning consumer (ToC), business (ToB), and employee (ToE) settings. By grounding tasks in real-world application ecosystems, the authors aim to measure how well agents can handle complex, multi-turn interactions, maintain state, and adapt to evolving user constraints.

Research Approach

The researchers formulated agent tasks as partially observable Markov decision processes (POMDPs), requiring agents to infer hidden states from partial observations and make sequential decisions. To ensure broad coverage, they synthesized 1,431 tasks through four distinct routes (DAG, DAG-S, Solver, and Program), ensuring that tasks reflect realistic workflows. The evaluation is supported by a ten-dimensional capability taxonomy—including planning, state management, and tool use—and an eight-factor difficulty framework. This allows for fine-grained diagnostic analysis of where agents fail, rather than just reporting a single success metric.

Key Findings

Evaluation of current frontier models, including Claude-Sonnet-5 and GPT-5.6-Sol, shows that even the most advanced systems struggle with the complexity of OmniaBench. The overall Pass@1 scores remain below 60%, highlighting persistent limitations in long-horizon planning, constraint maintenance, and adaptive error correction. The results suggest that while models have improved in basic tool invocation, they lack the robustness required for reliable, multi-step execution in heterogeneous, real-world environments.

As AI agents transition from simple text generators to autonomous systems capable of executing complex workflows, the need for diagnostic evaluation becomes critical. OmniaBench provides a standardized, scalable, and challenging framework to identify the specific capability boundaries of current models. By exposing weaknesses in planning and state tracking, this benchmark serves as a roadmap for future research into more reliable and capable agentic systems.