OSWORLD-PRO: PROCESS-BASED EVALUATION FOR COMPUTER USE AGENTS

Evaluating Computer-Use Agents (CUAs) has traditionally focused on final outcomes—whether an agent successfully produces a requested file or deliverable. While effective for simple tasks, this approach fails to provide transparency into long-horizon workflows where agents must navigate dozens of interdependent subgoals. Outcome-based evaluation cannot distinguish between an agent that fails early and one that nearly succeeds, nor can it identify why an agent fails (e.g., a keyboard input error versus a UI navigation mistake). OSWorld-Pro addresses these limitations by shifting the focus to process-based evaluation.

OSWorld-Pro consists of over 300 long-horizon tasks, comprising more than 2,800 sequentially dependent subgoals. The benchmark is grounded in over 67,000 human-annotated labels, providing a granular view of agent performance at every step. By breaking tasks into atomic, dependent subgoals, the researchers can monitor progress throughout the entire trajectory. The benchmark includes diverse applications and environments—ranging from rare Linux distributions to specialized software—to test the generalization capabilities of modern vision-language models.

Even state-of-the-art models struggle with OSWorld-Pro; for example, Claude Opus 5 achieves 75.7% accuracy, significantly lower than its performance on simpler benchmarks. The study identifies distinct failure modes, such as subgoal-irrelevant actions, where models perform dozens of unnecessary steps despite eventually completing the task. Furthermore, the researchers found that while some models excel at clicking, others struggle with basic coordinate selection or fail to recognize when a task is infeasible, leading to persistent, wasted effort. These insights provide a roadmap for developers to optimize agent harnesses and improve the efficiency of long-horizon task execution.