Niu Lian, Alan Chen, Zhehao Yu, Chengzhen Duan, Fazhan Liu, Hui Liu, Pei Fu, Jian Luan, Yaowei Wang, Shu-Tao Xia, Jinpeng Wang
5 min
Abstract
Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction. However, building multi-platform GUI agents remains challenging. On one hand, high-quality and executable cross-platform interaction trajectories are still scarce, and existing data often suffer from limited platform coverage. On the other hand, different platforms exhibit distinct interaction conventions, making joint or continual training prone to behavioral pattern mixing, platform-specific capability degradation, and catastrophic forgetting. To address these challenges, we construct Uni-GUI, a high-quality cross-platform GUI interaction dataset, and propose UI-MOPD, the first method that incorporates multi-teacher on-policy distillation into continual learning for GUI agents. UI-MOPD dynamically selects a platform-specific teacher according to the current environment and transfers platform-specific behavioral priors to a shared policy through platform-conditioned distillation, enabling adaptation to new platforms while preserving capabilities on existing ones. Experiments on OSWorld and MobileWorld show that UI-MOPD achieves task success rates of 38.2% and 12.0%, respectively, demonstrating its effectiveness in balancing cross-platform capability retention and new-platform adaptation. Project page: https://elispectre.github.io/UI-MOPD/.
Sam: How does the system actually measure whether the student is drifting away from the teacher? How does it know when to step in?
Alex: They use a technique called "on-policy distillation." Here's the practical picture: the student attempts a real task—say, navigating to a settings menu. At each step it takes, the system compares the student's choice against what the expert teacher would have done in that exact moment. If there's a meaningful gap, the student receives a correction signal. Crucially, this only happens on the steps the student is actually taking, not on every possible action in the universe.
Sam: That's a much more targeted approach than just making the student copy the teacher wholesale.
Alex: It is. And the technical tool they use to measure that gap is called a K3 estimator. Without going into the mathematics, think of it as a very precise ruler that measures how far apart two decision-making styles are. The smaller the gap, the less correction the student needs.
Sam: Does the system keep correcting the student forever, even when it's already doing well?
Alex: No, and this is where another piece of the design becomes important. They use something called "adaptive KL masking." Once the student demonstrates it has genuinely mastered a particular part of a task—it's making the right choices consistently—the system stops applying the correction signal for that part. It essentially says: you've got this, go explore. That freedom allows the agent to improve beyond simply mimicking the teacher.
Sam: So it's a staged process. Guided correction first, then independent exploration once the foundations are solid.
Alex: That's a good way to put it. And there's one more layer worth understanding. You asked earlier about the risk of the agent becoming too narrowly trained—good at a small slice of tasks but brittle outside of them.
Sam: Right—if you're only reinforcing a narrow set of examples, doesn't the agent just learn to game those specific cases?
Alex: The authors address this directly. They use a structured reward system that grades the agent on concrete, measurable goals—things like whether it clicked in precisely the right location on screen, or whether the text it produced matched what was required. Because these rewards are tied to real, verifiable outcomes rather than just "did it look like the teacher," the agent has to develop genuine capability, not just surface-level imitation.
Sam: So the combination of platform-specific coaching, targeted correction, and measurable rewards all work together to produce an agent that's actually reliable across different environments.
Alex: That's the central claim, and the evidence the authors present supports it. By keeping platform knowledge separate during training rather than blending it together, and by giving the agent room to grow once it's demonstrated competence, the system produces more stable performance across both desktop and mobile contexts. It's a considered solution to what turns out to be a surprisingly subtle problem.
Sam: It makes you realise how much invisible structure exists in the interfaces we use every day—structure that we've absorbed so naturally we don't even notice it, but that an AI has to be carefully taught to recognise.
Alex: That's exactly right. And understanding how to teach that structure—platform by platform, step by step—is what this research is working toward. Thanks for listening to ResearchPod.