Scaling GUI Agents with Visual State Transitions

Scaling GUI agents typically relies on trajectory fine-tuning, which is data-intensive and couples visual grounding, dynamics modeling, and task planning into a single, difficult-to-optimize objective. This paper investigates whether pretraining a model on atomic visual state transitions—independent of task instructions—can provide a more effective initialization for GUI agents.

The authors introduce State Transition Pretraining (STP), a two-stage training framework. In the first stage, a unified multimodal model is trained on transition tuples (current state, action, next state) using two complementary objectives:

1. Forward Dynamics: Predicting the next GUI screenshot given the current state and action, which helps the model internalize how the interface responds to interactions.
2. Inverse Dynamics: Predicting the action taken given two consecutive screenshots, which forces the model to identify relevant visual elements and action-specific changes.

Following this pretraining, the model is fine-tuned on standard trajectory data with task instructions. This approach allows the model to learn basic GUI mechanics and visual grounding before tackling complex, instruction-conditioned planning.

STP consistently outperforms models trained solely via direct trajectory fine-tuning across both desktop (AgentNetBench) and mobile (AndroidControl, GUIOdyssey) benchmarks. The authors demonstrate that STP provides a stable improvement regardless of the fine-tuning data volume, suggesting that the structured learning signals from dynamics modeling are highly complementary to task-level supervision. Empirical results show that scaling the amount of transition data leads to steady performance gains, and that joint optimization of both dynamics objectives is superior to single-objective training.

This work provides a scalable, annotation-free method to improve GUI agent performance. By decoupling the learning of interface dynamics from task planning, researchers can leverage vast amounts of unlabeled interaction data to build more robust agents, reducing the reliance on expensive, manually labeled trajectory datasets.