MAI-UI Team
5 min
Abstract
GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.
Alex: So it's not just memorizing solutions. It's learning how to recover when things go sideways.
Sam: Right. And to keep that learning process running at scale across hundreds of physical phones, they use what they call a "health-aware scheduler." Some phones will freeze, crash, or behave erratically. The scheduler automatically detects those problem devices, takes them out of rotation, and reroutes tasks to ones that are working. It's essentially building a stable production line out of inherently unpredictable hardware.
Alex: Like a floor manager — constantly watching which machines are running and which need to be sidelined.
Sam: That's a good way to put it. It keeps the whole pipeline running without needing a human to intervene every time a phone locks up. And for judging whether a task was actually completed correctly, they use a system called "AutoJudge" — five different vision models look at the screen after the agent finishes and vote on whether the result looks right. The paper notes it's quite accurate, though not perfect.
Alex: That's an interesting constraint — the system can't look inside the phone directly, so it has to watch the screen the way a human would.
Sam: Exactly. It's an automated observer rather than a direct internal check, which means there's some margin for error in how success gets measured. The paper is candid about that limitation.
Alex: What about desktop computers? Does any of this carry over?
Sam: That's still a work in progress. The researchers note that scaling the training to desktop environments is ongoing, and those results weren't ready for this report. So what we're seeing here is primarily the mobile side of the system.
Alex: Where does the field go from here, then?
Sam: The next significant challenge is speed. Right now, every single tap or swipe requires the model to look at the screen, process what it sees, and decide what to do — and that takes time. Making these agents feel genuinely responsive means reducing that per-step cost, either through faster processing or by letting the agent handle more work in the background.
Alex: And beyond speed?
Sam: The longer-term goal is handling complex, multi-day workflows — tasks that span different apps, different devices, and stretch over time. That requires much better memory, so the agent can keep track of where it left off. The researchers are also looking at what they call cross-domain training, where a single agent learns to handle mobile, web, and desktop tasks together, rather than being specialized for just one.
Alex: So the ambition is to move from a tool that follows instructions to something that manages the logistics of your digital life on your behalf.
Sam: That's the direction. The goal is a system that doesn't just execute a command when asked, but anticipates what you need and handles the background work. This research is a meaningful step toward that — though the paper is clear that significant work remains, particularly around safety and personalization. Thanks for listening to ResearchPod.