Madhumitha Venkatesan, Shicheng Wen, Jiajing Guo, Jorge Piazentin Ono, Liu Ren, Dongyu Liu
4 min
Modern vision-based GUI agents often operate as 'black boxes' where planning and adaptation occur internally. This creates a significant challenge for users: when an agent drifts from the intended task, users lack the visibility to understand why it failed or the tools to correct it without restarting the entire process. The authors ask: How can we design GUI automation systems that keep task structure visible, support precise interventions, and preserve progress during recovery?
To address these challenges, the authors developed Plover, a plan-centric GUI automation system. Unlike traditional agents that treat plans as transient internal states, Plover externalizes the task plan as a persistent, inspectable, and editable artifact. The system architecture separates the planner from the executor, allowing the agent to report its progress and proposed steps clearly.
Key features include:
The authors evaluated Plover through a formative study and a series of benchmark failure-case repairs. In a test of 26 failed autonomous tasks, the collaborative Plover interface allowed users to successfully repair 23 of them, with 17 reaching full completion. The results demonstrate that many GUI-agent failures are not terminal; they are often structurally repairable if the user is given the right tools to intervene at the correct moment. This research highlights that robust GUI automation is as much an interaction design problem as it is a model-performance problem.
Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.
Sam: And what happened?
Alex: The outcome was notable. Of those twenty-six previously failed tasks, seventeen became complete successes, and six became partial successes. Only three remained unresolved.
Sam: So the vast majority of tasks that had completely failed were recovered—just by making the plan visible and letting a person intervene. That's a meaningful shift.
Alex: It is. And what it suggests is that many of these failures aren't fundamental—they're not cases where the AI simply lacks the capability. They're what the researchers call "structurally repairable." The AI had a workable plan, but when one step went wrong, there was no mechanism to catch it. Add that mechanism, and the task can often be completed after all.
Sam: That reframes the whole problem, doesn't it? Instead of asking "how do we build an AI that never makes mistakes," the question becomes "how do we build a system where mistakes can be caught and corrected before they spiral?"
Alex: That's the central insight the paper offers. Plover suggests that the path to more reliable AI automation isn't necessarily a more powerful AI—it's a more transparent one. When the plan is visible, when a human can step in at the right moment, the collaboration between person and machine becomes more resilient than either working alone. Thanks for listening to ResearchPod.