Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

Computer-Use Agents (CUAs) are designed to operate graphical user interfaces (GUIs) based on natural-language instructions. A major bottleneck in training these agents via reinforcement learning (RL) is the lack of scalable, machine-readable reward signals. While vision-language models (VLMs) can act as autonomous evaluators to judge task completion, their feedback is inherently noisy. This paper investigates whether explicitly modeling and correcting this evaluator noise can enable stable and effective RL fine-tuning for desktop agents.

To address the challenge of noisy feedback, the authors treat the VLM-based evaluator as a binary reward channel with known error characteristics. They define a noise-corrected reward estimator that transforms the raw binary output of the evaluator into a statistically grounded, asymptotically unbiased reward signal. This estimator accounts for both false positives (incorrectly labeling a failure as a success) and false negatives (failing to detect a success). The authors then integrate this corrected reward into the Proximal Policy Optimization (PPO) algorithm to fine-tune agents across three major desktop environments: macOS, Windows, and Linux.

Empirical results across benchmarks like macOSWorld, Windows Agent Arena, and OSWorld demonstrate that the noise-corrected reward estimator consistently outperforms both zero-shot baselines and raw, uncorrected evaluator rewards. By explicitly correcting for evaluator bias, the agents achieved an average success rate improvement of 12.6 percentage points over zero-shot performance and 5.1 percentage points over raw evaluator fine-tuning. The findings suggest that autonomous evaluation is a viable, scalable reward mechanism for GUI agents, provided that the evaluator's error profile is explicitly modeled.

This work provides a practical path toward training general-purpose computer-use agents without the need for expensive manual labeling or brittle, task-specific heuristics. By enabling agents to learn from their own interactions in real-world desktop environments, this framework helps bridge the gap between current agent capabilities and the requirements for reliable, autonomous desktop automation.