Lychee Team
4 min
KnowAct-GUIClaw addresses the limitations of existing GUI agents, which often struggle with long-horizon tasks, cross-platform migration, and the inability to learn from past failures. The authors propose a "Know-Route-Act-Reflect" paradigm that separates the responsibilities of a host agent and a GUI subagent. The host agent manages high-level planning, user context, and tool selection, while the GUI subagent handles low-level visual interaction. By integrating a persistent memory system and a self-evolving skill library, the framework allows the agent to store successful interaction patterns and refine its behavior over time.
The framework operates through four distinct stages. In the Know stage, the agent retrieves relevant past experiences and user preferences to inform current tasks. The Route stage decomposes complex, multi-app requests into manageable subtasks with explicit input/output contracts. The Act stage executes these subtasks using a hybrid approach that combines standard GUI actions with faster, non-visual shortcuts like deep links or system intents. Finally, the Reflect stage analyzes the execution trajectory to distill new skills and update the memory store, ensuring that future tasks benefit from previous successes and failures.
This research is significant because it moves GUI agents beyond one-off, isolated task execution toward persistent, evolving personal assistants. By treating GUI automation as a collaborative effort between a reasoning host and an executor, the framework significantly reduces the computational overhead and fragility associated with monolithic GUI agents. The ability to transfer skills across different base models and platforms suggests a scalable path toward more reliable and efficient cross-device automation.
OpenClaw has emerged as a leading agent framework for complex task automation, yet it faces insufficient cross-platform GUI interaction support and a well-built self-evolution mechanism. These flaws limit its adaptation to diverse device ecosystems and prevent performance improvements through continuous learning from execution experience. To resolve these issues, we propose the Know Deeply, Act Perfectly paradigm for personal assistants, which holds that accumulated user interaction and task-running experience directly improve execution accuracy and efficiency, unifying cognitive comprehension and operational execution. Based on this paradigm, we introduce KnowAct-GUIClaw, a novel Know-Route-Act-Reflect framework designed to address OpenClaw's GUI manipulation deficits and break through its cross-platform and recursive self-improvement constraints. First, the host agent leverages accumulated interaction experience and task-relevant knowledge for long-horizon task decomposition and allocation (Know). Second, a pluggable GUI subagent with an experience-attributable memory system (Know) and self-evolving skill library (Act), enabling seamless cross-platform migration and fast-path integration. Especially, this framework continuously stores user profiles and feedback to improve the accuracy of task decomposition and tool calls. Extensive experiments across Android, iOS, HarmonyOS and Windows show that KnowAct-GUIClaw achieves superior efficiency, accuracy and cross-platform adaptability. Especially, the GUIClaw with open-source Kimi-2.6 models achieves the best performance (64.1%) on the long-horizon MobileWorld benchmark, beating all agentical frameworks and closed-source agentical models, e.g., Seed-2.0-Pro and GPT-5.5. Additionally, the knowledgeable memory and execution skills supported by our framework are transferable across diverse base models, improving by 8.5% with Kimi-2.6.
Sam: There is one significant limitation. Those saved skills need to be validated against reality. If an app updates its layout—moves a button, renames a menu—a shortcut that worked last week might lead the agent straight into a dead end. The system has to constantly check that its memory still matches what's actually on the screen.
Alex: So it builds a map, but has to keep verifying the map is still accurate.
Sam: That's a good way to put it. The researchers frame it like navigating a dark room with only a flashlight—you can only see what the beam hits at any given moment, so you have to constantly update your mental picture of where the furniture is. The agent faces the same problem: it never has a complete view of the interface, so it makes decisions based on partial information and revises as it goes.
Alex: How do the Host and Executor stay coordinated if they're handling such different jobs?
Sam: They communicate through what the paper calls a "Blackboard"—essentially a shared notepad where both sides can read and write. The Host posts the current goal and context; the Executor reads it, takes an action, and posts back what it found. It keeps both sides aligned without either one having to guess what the other meant.
Alex: So the whole system is a division of labor, a growing library of experience, and a constant reality-check loop, all working together.
Sam: That's the core insight. By separating the thinking from the doing, and by giving the system a way to genuinely learn from its own history rather than starting fresh each time, the agents become meaningfully more reliable. The skill validation problem is real and ongoing—but the direction is clear: toward systems that improve through experience rather than repeating the same mistakes indefinitely.
Alex: A thoughtful step forward. Thanks for walking us through it, Sam, and thanks to everyone listening to ResearchPod.