GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a research paper called Qwen-UI-Agent. The central puzzle is this: why do AI assistants work reasonably well in controlled lab tests, but fall apart the moment you hand them a real, glitchy smartphone?
Alex: That does seem like a significant gap. What's actually causing it?
Sam: The researchers call it the "simulation-to-reality gap." In a simulation, everything is predictable — apps always look the same, nothing interrupts, and every permission is granted. On a real device, you get random notifications, layouts that shift between app updates, and apps that behave differently depending on which phone you're using. An agent that only trained in that controlled sandbox tends to fall apart the moment something unexpected happens.
Alex: So how are they closing that gap? Is it just more practice, or something more fundamental?
Sam: More fundamental. They built a testing environment using over a hundred physical mobile devices — real phones, not simulations. But the core of their approach is something they call a "Hybrid GUI+CLI Action Space." Think of it like a Swiss Army knife for interacting with a device.
Alex: Different tools for different jobs?
Sam: Exactly. The "GUI" side — short for Graphical User Interface — is like the agent's eyes and hands. It taps buttons, swipes screens, and reads what's on the display, the same way you'd use your finger. The "CLI" side — Command Line Interface — is more like a power tool. Instead of tapping through a dozen menus to move a file, the agent can type a direct instruction to the operating system and get it done in one step. The agent picks whichever approach suits the moment — visual interaction when that's clearer, direct commands when that's quicker.
Alex: So it's not locked into one way of doing things. That seems like it would make it more robust.
Sam: It does, and more efficient too. On top of that, they've built what they call a "data flywheel" — an automated loop where the agent generates its own practice tasks, attempts them, identifies where it went wrong, and uses those failures to improve. It's essentially teaching itself, continuously.
Alex: That's a lot of self-directed learning. But what happens when a task is genuinely long and complicated — dozens of steps, maybe more?
Sam: That's where a technique called "online reinforcement learning" comes in. Rather than just copying examples it's seen before, the agent tries a sequence of actions, receives a signal telling it whether the outcome was correct, and gradually refines its strategy. Some of these task sequences run to a hundred steps or more. Specialized checking programs — called verifiers — act like an automated answer key, confirming at the end whether the agent actually succeeded.
Alex: So it's not just memorizing solutions. It's learning how to recover when things go sideways.
Sam: Right. And to keep that learning process running at scale across hundreds of physical phones, they use what they call a "health-aware scheduler." Some phones will freeze, crash, or behave erratically. The scheduler automatically detects those problem devices, takes them out of rotation, and reroutes tasks to ones that are working. It's essentially building a stable production line out of inherently unpredictable hardware.
Alex: Like a floor manager — constantly watching which machines are running and which need to be sidelined.
Sam: That's a good way to put it. It keeps the whole pipeline running without needing a human to intervene every time a phone locks up. And for judging whether a task was actually completed correctly, they use a system called "AutoJudge" — five different vision models look at the screen after the agent finishes and vote on whether the result looks right. The paper notes it's quite accurate, though not perfect.
Alex: That's an interesting constraint — the system can't look inside the phone directly, so it has to watch the screen the way a human would.
Sam: Exactly. It's an automated observer rather than a direct internal check, which means there's some margin for error in how success gets measured. The paper is candid about that limitation.
Alex: What about desktop computers? Does any of this carry over?
Sam: That's still a work in progress. The researchers note that scaling the training to desktop environments is ongoing, and those results weren't ready for this report. So what we're seeing here is primarily the mobile side of the system.
Alex: Where does the field go from here, then?
Sam: The next significant challenge is speed. Right now, every single tap or swipe requires the model to look at the screen, process what it sees, and decide what to do — and that takes time. Making these agents feel genuinely responsive means reducing that per-step cost, either through faster processing or by letting the agent handle more work in the background.
Alex: And beyond speed?
Sam: The longer-term goal is handling complex, multi-day workflows — tasks that span different apps, different devices, and stretch over time. That requires much better memory, so the agent can keep track of where it left off. The researchers are also looking at what they call cross-domain training, where a single agent learns to handle mobile, web, and desktop tasks together, rather than being specialized for just one.
Alex: So the ambition is to move from a tool that follows instructions to something that manages the logistics of your digital life on your behalf.
Sam: That's the direction. The goal is a system that doesn't just execute a command when asked, but anticipates what you need and handles the background work. This research is a meaningful step toward that — though the paper is clear that significant work remains, particularly around safety and personalization. Thanks for listening to ResearchPod.