Building capable visual web agents requires long-horizon reasoning, precise grounding, and robust interaction with dynamic real-world websites. Despite rapid progress, the strongest systems remain largely proprietary, while open agents still depend heavily on supervised post-training over large collections of curated web trajectories. This dependence creates a major scalability bottleneck: high-quality demonstrations are expensive to collect, and static datasets offer limited coverage of the diverse, ever-changing open web. Although online RL has shown promise for text-based agents, its potential for training visual web agents directly on live websites remains largely underexplored. In this paper, we introduce OpenWebRL, an open framework for training visual web agents with online multi-turn RL on real websites. OpenWebRL covers the full training pipeline, including scalable live-browser infrastructure, supervised initialization, multimodal context management, trajectory-level success judging, and efficient multi-turn policy optimization. Using this framework, we train OpenWebRL-4B, which establishes a new open-source state of the art on challenging live-web benchmarks. With only 0.4K initialization trajectories and 2.2K open-ended RL training tasks, OpenWebRL-4B achieves 67.0% success on Online-Mind2Web and 64.0% on DeepShop, outperforming prior open agents of similar or larger scale and remaining competitive with proprietary systems including OpenAI CUA and Gemini CUA. Beyond strong benchmark performance, we systematically study the key design choices that make online RL effective for visual web agents, and analyze how RL improves agentic reasoning. Overall, our work offers a practical path toward building more capable, reproducible, and cost-efficient open web agents. We will release our training data, models, and code to support future research.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a framework called OpenWebRL — and the central puzzle it tackles is this: how do you train an AI to use a web browser — things like searching for flights or filling out forms — without needing enormous, expensive collections of examples recorded by humans?
Alex: So instead of the AI copying what humans have already done, it's learning by actually doing things itself?
Sam: Exactly. Most current AI agents that work on the web are trained on static datasets — think of them like textbooks written once and never updated. Those textbooks go out of date fast, and building them in the first place is costly. This research proposes a different approach: let the agent learn through trial and error on the live web, in real time.
Alex: If it's learning live, how does it know whether it's actually doing a good job?
Sam: That's the core challenge. They use something called trajectory-level judging. Instead of checking whether any single click was correct — like marking one answer right or wrong on a quiz — a judge model watches the agent's entire sequence of actions and evaluates whether the overall goal was achieved. Did the flight get booked? Did the right product end up in the cart? The whole journey gets graded, not just individual steps.
Alex: And that judgment is what drives the learning?
Sam: Right. The specific technique is called Group-Relative Policy Optimization. Here's how to think about it: imagine a student trying five different approaches to solve the same problem. Instead of just getting a pass or fail on each one, the system compares all five attempts against each other and rewards the strategies that worked better than average. The agent learns which approaches tend to succeed — without needing a separate, complex system constantly estimating the value of every individual action.
Alex: That does sound cleaner. But surely you can't just throw an untrained model at a live website and expect it to figure things out?
Sam: You're right — and that's where a "warm start" comes in. Before any live trial-and-error begins, the model is trained on a small, carefully chosen set of tasks. Think of it like giving someone a driving manual before they sit behind the wheel. Without that foundation, the agent doesn't even know how to interact with a browser in a basic way, so the trial-and-error phase produces nothing useful. The initial training gives it just enough grounding to start exploring meaningfully.
Alex: So the first phase teaches it the basics, and the second phase is where it actually develops judgment?
Sam: That's a good way to put it. And there's another important piece: the agent doesn't just read text on a page — it looks at screenshots. The system uses AI models that can look at an image and understand what's on it. So when the agent sees a webpage, it's not reading raw code; it's interpreting a visual layout, the same way a person would. That lets it find buttons, read menus, and understand where things are on the screen.
Alex: Which matters a lot on the real web, where pages aren't neatly structured.
Sam: Exactly. And the live web introduces another problem: things break. Pages crash, pop-ups appear, buttons stop responding. To handle that, the researchers built what they call a sandboxed environment — each attempt the agent makes runs in its own isolated container, like a separate practice room, so if one session crashes, it doesn't corrupt the rest of the training. And when an action fails, the browser sends back an error signal, which gives the agent a concrete reason to try a different approach.
Alex: So it's not just guessing — it's getting real feedback and adjusting.
Sam: Precisely. And all of this was used to train a model that's relatively compact by modern standards. Despite its smaller size, it performs competitively with much larger, closed-source systems on standard web-task benchmarks. The paper's suggestion is that a well-designed training process can matter more than simply making the model bigger.
Alex: That's a meaningful finding. It implies you don't necessarily need a huge proprietary system to build something capable.
Sam: That's the broader point the researchers are making. By moving to this live, trial-and-error approach, they've created a path for the wider research community to build capable web agents without depending on expensive private datasets or massive infrastructure — making this kind of AI more accessible and reproducible.
Alex: It's a thoughtful piece of work. The idea that you can build a capable agent by letting it learn from real experience — with the right scaffolding — rather than simply feeding it more and more pre-recorded data, points to a meaningful shift in how people are thinking about this problem. Thanks for listening to ResearchPod.