Training Open Models for Agentic Phone Use | Zhengyang Tang et al. | ResearchPod

Transcript: Training Open Models for Agentic Phone Use

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a study about how to make AI agents better at using smartphones. Sam, what's the core puzzle here?

Sam: We're discussing a project called PhoneBuddy. The central challenge is that training an AI to use a phone is genuinely difficult — real devices are messy, slow to reset, and can accidentally change things like your settings or messages. The researchers want to know how to train these models to be reliable without relying solely on slow, real-world testing.

Alex: So the paper is asking how to bridge the gap between a perfect, controlled practice environment and the unpredictable reality of an actual smartphone?

Sam: Exactly. You have two extremes. Real phones are authentic but hard to manage for training purposes. Mock environments — essentially simplified, digital replicas of apps — are fast and easy to reset, but they don't always behave the way real apps do. The researchers argue you shouldn't choose just one. They propose a method that combines both.

Alex: That reminds me of how you'd train a pilot. You wouldn't skip the flight simulator, but you also wouldn't put a student straight into a real cockpit. How exactly do they combine the two?

Sam: They use a technique called "reinforcement learning." Imagine training a dog with treats — when the dog sits on command, it gets a reward; when it ignores you, it doesn't. The AI works the same way. Every time it takes a correct action on the phone, it receives a reward signal. Every time it fails, it gets a penalty. The system gradually learns which actions lead to success.

Alex: So the AI gets to practice in the safe, fake environment over and over, and then gets real-world experience to learn how actual apps behave?

Sam: That's it. They call this approach "mixed reinforcement learning." In the mock environment, the model can reset instantly after a mistake and practice the same navigation step hundreds of times without consequence. But the mock version is missing something important — real apps have side effects. Logging into an actual account, triggering a notification, changing a setting. Those things only happen on a real phone, so that's where the second phase of training comes in.

Alex: And why does that distinction matter so much? Couldn't the mock environment just simulate those side effects?

Sam: You could try, but the gap between simulated and real behavior is surprisingly hard to close. A mock environment might show a button in the right place, but the real app might load slowly, display a pop-up, or respond differently depending on account state. The AI trained only on mock environments tends to be brittle — it works well in the practice room but stumbles the moment something unexpected appears on the real screen.

Training Open Models for Agentic Phone Use

Zhengyang Tang, Xin Lai, Pengyuan Lyu, Xinyuan Wang, Tianyi Bai, Chenxin Li, Yiduo Guo, Huawen Shen, Yuxuan Liu, Junyi Li, Zhengyao Fang, Yang Ding, Yi Zhang, Weinong Wang, Xingran Zhou, Liang Wu, Fei Tang, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Benyou Wang, Ji-Rong Wen, Rui Yan, Chengquan Zhang, Han Hu

Jun 22, 2026·4:02·arxiv·Original Paper

PhoneBuddyAgentic Phone UsePhoneWorldReinforcement Learning (RL)Supervised Fine-TuningReal-app RL

Listen

4 min

Briefing

Research Context and Approach

Training agents for mobile devices is notoriously difficult because real-world phone environments are stateful, slow, and prone to side effects, making them difficult to reset or verify automatically. While mock environments offer scalability, they often fail to capture the nuances of real-device behavior. PhoneBuddy addresses this by introducing a training recipe that leverages both real-app environments and a reconstructed mock-app environment called PhoneWorld. PhoneWorld reconstructs runnable mock apps from real GUI structures, allowing for scalable, resettable, and automatically checked interaction. The researchers compare three training stages: supervised fine-tuning (SFT), real-app reinforcement learning (RL), and a mixed RL approach that incorporates both real-app and mock-app data.

Main Findings

The study demonstrates that combining real-app and mock-app training is superior to using either in isolation. On a 150-task human evaluation conducted on real phones, the task success rate improved from 36.67% after SFT to 40.67% after real-app RL, and further to 45.33% after mixed RL. A similar trend was observed on the AndroidWorld benchmark, where success rates rose from 60.3% to 77.2% and finally to 83.2%. These results suggest that mock-app training acts as a powerful, complementary source of interaction signal rather than a replacement for real-world execution.

Why It Matters

This work highlights that the primary bottleneck for mobile agents is not just perception or grounding, but the ability to train models that can reliably navigate the complexities of real-world software. By showing that mock environments can be effectively used to augment real-device training, the authors provide a scalable path forward for developing more capable phone agents. However, the persistent difficulty with long-horizon cross-app workflows indicates that future research must focus on better state tracking and information handoff between applications.

Key Terms and Definitions

PhoneBuddy — A training recipe and open-model line that combines real-app reinforcement learning with a mock-app environment to improve agentic phone use.
PhoneWorld — A mock-app environment that reconstructs runnable Android apps from real GUI usage structures to enable scalable and verifiable agent training.
Real-app RL — A reinforcement learning stage where the agent operates authentic apps on physical devices to optimize for real-world task completion.
Mixed RL — A training strategy that performs reinforcement learning across both real-app and mock-app environments to balance high-fidelity execution with scalable interaction signals.
Rubric-based model judging — A method used in the real-app environment where a secondary model scores interaction traces against task-specific rubrics to provide a reward signal for reinforcement learning.

Abstract

Phones are becoming an important execution surface for general-purpose agents, but training open models for reliable phone use remains difficult because the environment that matters at deployment, real devices running real apps, is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. We present PhoneBuddy, a training recipe and open-model line for agentic phone use that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL across both environments. Across a 150-task human evaluation on real phones spanning apps, mini-apps, and cross-app workflows, task success rate improves from 36.67\% after supervised fine-tuning to 40.67\% after real-app RL and 45.33\% after mixed RL. On AndroidWorld, the same progression rises from 60.3\% to 77.2\% to 83.2\%. These results show that mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction. The gains are strongest on app and mini-app tasks, while long-horizontal cross-app workflows remain an important open challenge.

Training Open Models for Agentic Phone Use: Research Paper Podcast