Zikai Xiao, Jianhong Tu, Chuhang Zou, Yuxin Zuo, Zhi Li, Peng Wang, Bowen Yu, Fei Huang, Junyang Lin, Zuozhu Liu
8 min
Abstract
Web agents require massive trajectories to generalize, yet real-world training is constrained by network latency, rate limits, and safety risks. We introduce \textbf{WebWorld} series, the first open-web simulator trained at scale. While existing simulators are restricted to closed environments with thousands of trajectories, WebWorld leverages a scalable data pipeline to train on 1M+ open-web interactions, supporting reasoning, multi-format data, and long-horizon simulations of 30+ steps. For intrinsic evaluation, we introduce WebWorld-Bench with dual metrics spanning nine dimensions, where WebWorld achieves simulation performance comparable to Gemini-3-Pro. For extrinsic evaluation, Qwen3-14B trained on WebWorld-synthesized trajectories improves by +9.2\% on WebArena, reaching performance comparable to GPT-4o. WebWorld enables effective inference-time search, outperforming GPT-5 as a world model. Beyond web simulation, WebWorld exhibits cross-domain generalization to code, GUI, and game environments, providing a replicable recipe for world model construction.
Alex: So a clean blueprint of the controls. And they had to filter this data somehow to keep only the good stuff, right?
Sam: Yes, a two-step filter first checks sites for accessibility and blocks unsafe content like gambling pages using rules and keyword scans. Trajectories get pruned if actions cause no change or exceed length limits, all with simple rules to avoid bias. They then enrich the data by converting trajectories into formats like text summaries or other structures, plus adding general chat examples to prevent forgetting basic skills.
Alex: Rule-based cleaning, then mixing in variety. That keeps quality high without over-relying on judgment calls. Training follows a two-part plan—how does that work?
Sam: First, the model learns basic page changes from the full million-plus trajectories. Second, they add a small set—about a thousand—of step-by-step reasoning examples, where the model describes why an action leads to a certain outcome before predicting it. This 'Chain-of-Thought' injection activates deeper thinking patterns already in the model. The paper shows this small dose works better than ten times more without the base training.
Alex: Oh—so reasoning builds on the dynamics they've already learned. Does this synthetic data actually help real agents perform better?
Sam: It does. They generated new trajectories using an abstract-then-concretize method: start with vague goals in the simulator, run them, then make specific versions and keep successes. Fine-tuning smaller models on this boosted success rates on benchmarks—a clear improvement from safer, scaled practice.
Alex: Notable gains there. How crucial was that small set of reasoning examples—did they test varying amounts?
Sam: They did an ablation study, testing different quantities of those step-by-step reasoning samples on the model's prediction accuracy. After the initial training on over a million basic interaction examples, just a thousand reasoning samples boosted the total score notably. The paper suggests this works because the base training first teaches the model real-world patterns, then the reasoning data activates its built-in thinking skills without overwhelming it.
Alex: So a foundation first, then targeted activation. And larger models? Does throwing more power at it keep improving things?
Sam: Yes, they trained across six model sizes and found a scaling law: as compute increases, prediction errors drop predictably in a power-law pattern, like how more practice steadily sharpens a skill without plateauing soon. Extrapolating suggests even bigger models could improve substantially further. No signs of hitting a wall yet.
Alex: Steady gains from size and compute. Does this setup transfer to other digital worlds, like apps or games?
Sam: It does. Fine-tuning on examples from areas like code editing or desktop interfaces gave about twice the performance of baselines on average, using the same prediction checks. This shows the approach adapts well across environments.
Alex: Practical, but any downsides in the predictions?
Sam: The paper notes limitations: the simulator sometimes shows sycophancy bias, generating overly optimistic results that favor the agent's choices. It also struggles with creating high-quality detailed content, like full articles. These are areas for future work.
Alex: Those biases make sense for a simulator pushing boundaries. But to get solid predictions despite that, they tuned the training carefully—why split it into stages?
Sam: They used a two-stage setup to build skills step by step. In the first stage, the model practices predicting what a webpage looks like after an action, using over a million real examples—think of it as drilling basic patterns, like how clicking a button usually loads a new form. This teaches the core rules of how web pages change without any extra thinking. The second stage adds a small batch of about a thousand examples where the model first explains its reasoning before predicting, like pausing to say 'clicking here should show results because...' This activates deeper cause-and-effect understanding already lurking in the model.
Alex: Foundation of raw patterns, then sparking the 'why' with targeted examples. Did they tweak settings between stages?
Sam: Starting with basics avoids confusing the model early on; jumping straight to reasoning without that groundwork leads to weaker results, as their tests showed. For stage one, they ramped up data flow with a higher learning speed—twice as fast as stage two—and packed examples tightly to cover ground quickly. Stage two slows things down with a lower speed and extra safeguards, like masking old history so it focuses only on reasoning parts, preventing it from forgetting the dynamics it learned first.
Alex: That conservative tweak in stage two sounds key to stability. Pulling it all together, what stands out as the real contribution here?
Sam: The core advance is a scalable way to gather and process over a million real-web trajectories safely, using structured page summaries like the A11y Tree—essentially a compact list of interactive elements pulled via Playwright. This feeds a two-stage training that first builds knowledge of page changes, then injects reasoning patterns with targeted Chain-of-Thought examples, proving essential for accurate predictions. It enables simulators for any digital interface, from web to apps, allowing safe, large-scale agent practice without live-site risks. The paper highlights sycophancy and content generation limits as areas for future work, but shows meaningful gains in generalization.
Alex: Balanced scaling with known edges to sharpen. This feels like a solid step for training agents practically.
Sam: It is. By focusing on quality trajectories through filtering, augmentation into varied formats, and alignment with model priors, it sets a path for universal digital simulators. That's the measured progress from this work. Thanks for the discussion.