Imene Kerboua, Fatemeh Pesaran Zadeh, Xing Han Lù, Weijian Qi, Alexander Miller, Junyi Song, Yunjia Tian, Dongjin Kang, Seyeon Choi, Marzia Nouri, Ewen Gueguen, Matteo Boglioni, Fengyuan Liu, Zeyi Liao, Mengqi Yuan, Yue Li, Alexandre Lacoste, Alexandre Drouin, Spandana Gella, Huan Sun, Gunhee Kim, Siva Reddy
7 min
Abstract
Web agents powered by large language and vision-language models are increasingly applied to realistic browser work that spans heterogeneous applications, multimodal content, and stateful workflows. However, existing reproducible web-agent benchmarks cover only a small number of web applications drawn from a few software categories, and restrict modality to text and vision. Live benchmarks broaden site coverage but sacrifice reproducibility, since pages and data drift between runs. Moreover, existing benchmarks do not meaningfully evaluate whether agents can understand and use audio and video content embedded within web tasks. To address these gaps, we introduce WebArena-Pro, a benchmark comprising 300 tasks across 20 self-hosted web applications in six domain categories, spanning distinct interface conventions, workflows, and data models. Across the evaluated agents, the best performance is achieved by both Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 at 62.3%, followed by OpenCode with Kimi 3.1 Pro at 60.7%. The strongest GenericAgent configurations, Gemini 3.1 Pro omni and Gemma4 31B, both achieve 30.3%. Among reproducible, human-curated web agent benchmarks, WebArena-Pro provides the broadest application coverage and the most comprehensive multimodal support to date. The benchmark treats audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. WebArena-Pro runs each task in isolation and supports reproducible, parallel evaluation. Tasks are authored through a dedicated annotator interface, filtered by LLM-assisted triage, and finally validated by humans before release. Project page: https://webarena-pro.github.io
Sam: That controls contamination between tasks. But the other claim is about media. What changes beyond giving an agent a screenshot?
Alex: The agent receives an accessibility tree, a structured description of page elements, alongside a screenshot. WebArena-Pro adds audio and video sources to that description. Dedicated tools can turn speech into text or sample frames from a video.
Sam: A transcript won’t tell you everything visible in a video. Do they show a task where that distinction actually matters?
Alex: They do. One task asks whether a video about Zener diodes shows any soldering. The agent opens the video and requests a transcript, but the transcript doesn’t mention soldering. It then extracts frames that show a soldering iron in use, and answers yes.
Sam: So the useful information isn’t necessarily in the page text, or even in the speech. The agent has to choose another way to inspect the content.
Alex: That’s what the example demonstrates. The benchmark also supports models that accept audio and video directly, rather than only transcripts and still images. Unlike VideoWebArena, which adds external tutorial videos to existing websites, this benchmark puts media inside the applications themselves.
Sam: How do they decide whether the agent succeeded? A plausible final answer could hide a lot of unfinished work.
Alex: Each task has a human-written rubric, a checklist of required actions and outcomes. Some items check whether the agent took a required step. Others check whether the final website state or answer matches the request.
Sam: That sounds useful, but who checks the checklist? Another model could introduce its own errors.
Alex: They combine checks. Some inspect stored application data, while others match visited addresses or actions. A language-model judge handles criteria that need interpretation and can recognize unexpected valid solutions. A task succeeds only when every required criterion passes.
Sam: So this isn’t entirely automatic verification against a database. Some results depend on a model’s judgment, even though the criteria come from people.
Alex: Yes. Human validation applies to task construction too. The authors write the tasks, automated triage filters candidates, and another annotator checks feasibility and scoring. The triage also discards tasks an agent solves correctly on its first attempt, so this is deliberately filtered evaluation work.
Sam: With that setup in mind, what’s the headline result, and what opportunity did each system have to finish?
Alex: The top overall success rate is about 62 percent, shared by Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5. Each task allows at most 50 browser actions. Success means completing the entire rubric, not merely making useful progress.
Sam: In other words, even the leaders leave a substantial share unfinished under that action limit. Where does performance break down most consistently?
Alex: Tasks spanning multiple applications are consistently harder than tasks within one application. That holds across the evaluated configurations, although the size of the drop varies. No configuration leads outright across every application domain, either.
Sam: What about the media claim? An agent might answer a video question from its title or thumbnail without inspecting the video.
Alex: The authors address that with a media-access experiment. They compare agents with media tools against versions without them. They also audit a subset of 43 tasks judged to require the actual media content, rather than titles, metadata, or thumbnails alone.
Sam: Does removing access make a meaningful difference on that subset, rather than just on the easier-to-guess questions?
Alex: For Claude Code with Opus 5.5, success rises from about 14 percent without media tools to about 49 percent with them. That supports the importance of content access in this setting. But the subset was selected after the fact, so I’d treat it as targeted evidence, not a separate pre-established test.
Sam: The overall ranking compares different models running inside different software. Can we tell whether the stronger model or the surrounding system deserves the credit?
Alex: Not from these comparisons alone. The authors explicitly flag that limitation. A harness is the software connecting a model to its observations and tools, and these systems use different harnesses and configurations.
Sam: Then equal action limits don’t necessarily make the resource comparison fair. One action could involve more computation or more tool use.
Alex: The paper says that too. Token use, observation frequency, native tool use, and latency differ across agents. Controlled comparisons across harnesses and repeated runs would strengthen claims about efficiency and reliability. And these are controlled applications, not evidence that the same success rates hold on changing production websites.
Sam: Who should make time for the full document, and where should they start?
Alex: Researchers building browser agents or designing evaluations should read it. Start with Tasks and Action Space, then Evaluation, to understand what agents can observe and what counts as success. Read Media Tools Ablation for the clearest test of why media access matters, and keep the limitations beside the rankings.
Sam: And what should everyone else remember after the system names and percentages fade?
Alex: An agent’s success depends on the workflow and the evidence it can access, not just the model behind it. WebArena-Pro makes those demands testable under repeatable conditions, while showing how much remains unfinished.
Sam: Thanks for listening.