Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong
4 min
Computer-using agents (CUAs) are increasingly capable of performing complex tasks across web, mobile, and desktop environments. To scale the development of these agents—through reinforcement learning, data curation, and performance evaluation—researchers need a reliable way to verify whether an agent successfully completed a task. Because human-written verifiers are brittle and manual annotation is too slow, the field has adopted vision-language models (VLMs) as automated judges. This paper investigates whether these VLM judges are actually reliable enough to serve as the foundation for agent development.
The authors introduce OSReward, a high-quality, cross-platform benchmark designed to evaluate VLM judges. Unlike previous efforts that reuse existing, potentially noisy trajectory data, OSReward is built from scratch. The researchers created realistic, lived-in environments across four platforms (Web, Windows, Ubuntu, and Android), populated them with everyday files and user states, and generated 1,019 human-verified trajectories. These trajectories were produced by diverse agent backbones, ensuring a mix of successes and failures. The benchmark includes three distinct views: the full set for general evaluation, OSReward-Hard for diagnosing performance on challenging cases, and OSReward-Multi for fine-grained alignment and efficiency scoring.
The study evaluates 27 state-of-the-art VLM judges, revealing a critical "leniency bias": models are easily fooled by agents that claim to have completed a task but have actually failed. While top-tier commercial models perform adequately on simple tasks, their accuracy drops significantly on the OSReward-Hard set. Furthermore, the most reliable judges are too expensive for large-scale training or evaluation pipelines. To address this, the authors curated OS-Shepherd-100K, a large-scale, reasoning-annotated corpus of trajectory judgments. They used this data to train OS-Shepherd (9B and 35B), open-source reward models that provide reliable, low-cost signals, matching the performance of frontier commercial models at 30–60% lower cost.
This work provides a standardized, rigorous framework for assessing the "judge" models that are becoming the backbone of agent training. By exposing the systematic failure modes of current VLMs and providing an open-source, cost-effective alternative, the authors enable the research community to build more robust and reliable autonomous agents without relying on expensive, proprietary black-box models.
Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.
Sam: It's like training a teacher to stop reading the cover letter and actually mark the assignment. Did that make a measurable difference?
Alex: The evidence suggests it did. OS-Shepherd matched the accuracy of the top commercial judges while costing somewhere between thirty and sixty times less to run. That gap matters, because it means reliable evaluation becomes practical for research teams who can't afford the high-end models.
Sam: So it's not just about accuracy in isolation—it's about making honest evaluation accessible. Where does the research point from here?
Alex: The paper highlights what it calls "hard cases"—situations where the agent confidently claims success but has actually failed. Those are the moments where leniency bias does the most damage, and the researchers suggest that standardizing how we handle those cases is the clearest path forward. Get those right, and you start breaking the false-success cycle at its source.
Sam: And if you can identify exactly where the judges are being fooled, you stop the agents from learning that a convincing story is a substitute for actually doing the work. That's a meaningful step toward systems we can genuinely trust.
Alex: It is. The paper's contribution is less about a single dramatic fix and more about building the infrastructure for honest evaluation—something the field needs before these systems take on more responsibility. Thanks for listening to ResearchPod.