Niu Lian, Alan Chen, Zhehao Yu, Chengzhen Duan, Fazhan Liu, Hui Liu, Pei Fu, Jian Luan, Yaowei Wang, Shu-Tao Xia, Jinpeng Wang
5 min
As GUI agents evolve from single-platform tools to cross-platform assistants, they face a significant challenge: how to adapt to heterogeneous environments (like desktop and mobile) without losing the specific interaction conventions of each. Naive approaches like mixed supervised fine-tuning or model merging often lead to an averaged, suboptimal policy or the catastrophic forgetting of previously learned platform-specific behaviors. This paper investigates how to train a single, unified GUI agent that can effectively navigate diverse platforms while retaining specialized interaction expertise.
To address these challenges, the authors introduce UI-MOPD (Multi-Platform On-Policy Distillation). The training process consists of two stages. First, they construct Uni-GUI, a high-quality dataset of cross-platform interaction trajectories, and use it to train platform-specific expert teachers for desktop and mobile environments. Second, they employ multi-teacher on-policy distillation (MOPD). During online reinforcement learning, the student policy samples rollouts from its current state. A platform-conditioned router then selects the appropriate teacher—desktop or mobile—to provide guidance via a reverse KL divergence penalty. This ensures that the student model aligns its behavior with the correct platform-specific interaction patterns, preventing the "averaging" effect common in joint training.
UI-MOPD demonstrates superior performance compared to traditional integration strategies. In experiments on the OSWorld and MobileWorld benchmarks, the model achieved task success rates of 38.2% and 12.0%, respectively. These results represent significant relative improvements over base models and outperform alternative methods like model merging or mixed supervised fine-tuning. Crucially, the approach successfully balances adaptation to new environments with the retention of existing capabilities, as evidenced by the model's performance on static GUI grounding and understanding tasks, where it maintains or exceeds the base model's accuracy while other methods show clear degradation.
This work provides a scalable, effective framework for building generalist GUI agents. By using on-policy distillation as a behavioral anchor, the authors offer a solution to the "seesaw effect" in multi-task learning, where improving performance on one platform typically degrades it on another. This approach paves the way for more robust, versatile digital agents capable of handling complex, cross-platform workflows in real-world settings.
Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction. However, building multi-platform GUI agents remains challenging. On one hand, high-quality and executable cross-platform interaction trajectories are still scarce, and existing data often suffer from limited platform coverage. On the other hand, different platforms exhibit distinct interaction conventions, making joint or continual training prone to behavioral pattern mixing, platform-specific capability degradation, and catastrophic forgetting. To address these challenges, we construct Uni-GUI, a high-quality cross-platform GUI interaction dataset, and propose UI-MOPD, the first method that incorporates multi-teacher on-policy distillation into continual learning for GUI agents. UI-MOPD dynamically selects a platform-specific teacher according to the current environment and transfers platform-specific behavioral priors to a shared policy through platform-conditioned distillation, enabling adaptation to new platforms while preserving capabilities on existing ones. Experiments on OSWorld and MobileWorld show that UI-MOPD achieves task success rates of 38.2% and 12.0%, respectively, demonstrating its effectiveness in balancing cross-platform capability retention and new-platform adaptation. Project page: https://elispectre.github.io/UI-MOPD/.
Sam: How does the system actually measure whether the student is drifting away from the teacher? How does it know when to step in?
Alex: They use a technique called "on-policy distillation." Here's the practical picture: the student attempts a real task—say, navigating to a settings menu. At each step it takes, the system compares the student's choice against what the expert teacher would have done in that exact moment. If there's a meaningful gap, the student receives a correction signal. Crucially, this only happens on the steps the student is actually taking, not on every possible action in the universe.
Sam: That's a much more targeted approach than just making the student copy the teacher wholesale.
Alex: It is. And the technical tool they use to measure that gap is called a K3 estimator. Without going into the mathematics, think of it as a very precise ruler that measures how far apart two decision-making styles are. The smaller the gap, the less correction the student needs.
Sam: Does the system keep correcting the student forever, even when it's already doing well?
Alex: No, and this is where another piece of the design becomes important. They use something called "adaptive KL masking." Once the student demonstrates it has genuinely mastered a particular part of a task—it's making the right choices consistently—the system stops applying the correction signal for that part. It essentially says: you've got this, go explore. That freedom allows the agent to improve beyond simply mimicking the teacher.
Sam: So it's a staged process. Guided correction first, then independent exploration once the foundations are solid.
Alex: That's a good way to put it. And there's one more layer worth understanding. You asked earlier about the risk of the agent becoming too narrowly trained—good at a small slice of tasks but brittle outside of them.
Sam: Right—if you're only reinforcing a narrow set of examples, doesn't the agent just learn to game those specific cases?
Alex: The authors address this directly. They use a structured reward system that grades the agent on concrete, measurable goals—things like whether it clicked in precisely the right location on screen, or whether the text it produced matched what was required. Because these rewards are tied to real, verifiable outcomes rather than just "did it look like the teacher," the agent has to develop genuine capability, not just surface-level imitation.
Sam: So the combination of platform-specific coaching, targeted correction, and measurable rewards all work together to produce an agent that's actually reliable across different environments.
Alex: That's the central claim, and the evidence the authors present supports it. By keeping platform knowledge separate during training rather than blending it together, and by giving the agent room to grow once it's demonstrated competence, the system produces more stable performance across both desktop and mobile contexts. It's a considered solution to what turns out to be a surprisingly subtle problem.
Sam: It makes you realise how much invisible structure exists in the interfaces we use every day—structure that we've absorbed so naturally we don't even notice it, but that an AI has to be carefully taught to recognise.
Alex: That's exactly right. And understanding how to teach that structure—platform by platform, step by step—is what this research is working toward. Thanks for listening to ResearchPod.