Rui Yang, Qianhui Wu, Yuxi Chen, Hao Bai, Wenlin Yao, Hao Cheng, Baolin Peng, Huan Zhang, Tong Zhang, Jianfeng Gao
4 min
Abstract
Building capable visual web agents requires long-horizon reasoning, precise grounding, and robust interaction with dynamic real-world websites. Despite rapid progress, the strongest systems remain largely proprietary, while open agents still depend heavily on supervised post-training over large collections of curated web trajectories. This dependence creates a major scalability bottleneck: high-quality demonstrations are expensive to collect, and static datasets offer limited coverage of the diverse, ever-changing open web. Although online RL has shown promise for text-based agents, its potential for training visual web agents directly on live websites remains largely underexplored. In this paper, we introduce OpenWebRL, an open framework for training visual web agents with online multi-turn RL on real websites. OpenWebRL covers the full training pipeline, including scalable live-browser infrastructure, supervised initialization, multimodal context management, trajectory-level success judging, and efficient multi-turn policy optimization. Using this framework, we train OpenWebRL-4B, which establishes a new open-source state of the art on challenging live-web benchmarks. With only 0.4K initialization trajectories and 2.2K open-ended RL training tasks, OpenWebRL-4B achieves 67.0% success on Online-Mind2Web and 64.0% on DeepShop, outperforming prior open agents of similar or larger scale and remaining competitive with proprietary systems including OpenAI CUA and Gemini CUA. Beyond strong benchmark performance, we systematically study the key design choices that make online RL effective for visual web agents, and analyze how RL improves agentic reasoning. Overall, our work offers a practical path toward building more capable, reproducible, and cost-efficient open web agents. We will release our training data, models, and code to support future research.
Sam: That's a good way to put it. And there's another important piece: the agent doesn't just read text on a page — it looks at screenshots. The system uses AI models that can look at an image and understand what's on it. So when the agent sees a webpage, it's not reading raw code; it's interpreting a visual layout, the same way a person would. That lets it find buttons, read menus, and understand where things are on the screen.
Alex: Which matters a lot on the real web, where pages aren't neatly structured.
Sam: Exactly. And the live web introduces another problem: things break. Pages crash, pop-ups appear, buttons stop responding. To handle that, the researchers built what they call a sandboxed environment — each attempt the agent makes runs in its own isolated container, like a separate practice room, so if one session crashes, it doesn't corrupt the rest of the training. And when an action fails, the browser sends back an error signal, which gives the agent a concrete reason to try a different approach.
Alex: So it's not just guessing — it's getting real feedback and adjusting.
Sam: Precisely. And all of this was used to train a model that's relatively compact by modern standards. Despite its smaller size, it performs competitively with much larger, closed-source systems on standard web-task benchmarks. The paper's suggestion is that a well-designed training process can matter more than simply making the model bigger.
Alex: That's a meaningful finding. It implies you don't necessarily need a huge proprietary system to build something capable.
Sam: That's the broader point the researchers are making. By moving to this live, trial-and-error approach, they've created a path for the wider research community to build capable web agents without depending on expensive private datasets or massive infrastructure — making this kind of AI more accessible and reproducible.
Alex: It's a thoughtful piece of work. The idea that you can build a capable agent by letting it learn from real experience — with the right scaffolding — rather than simply feeding it more and more pre-recorded data, points to a meaningful shift in how people are thinking about this problem. Thanks for listening to ResearchPod.