Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang, Jiahao Li, Bin Li, Yan Lu
7 min
Abstract
Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination.
Sam: So the model gets a readable working view, while the interface retains the detail needed to act. What reduces the repeated back-and-forth?
Alex: The agent can combine known actions into a batch. It can also insert a wait for an expected state before continuing. The same call returns observed changes, called an AX diff, rather than requiring a separate inspection every time.
Sam: A successful tool call does not necessarily mean the application did what you wanted. How does the interface handle that distinction?
Alex: It separates input delivery from evidence of an effect. If an action fails or a wait expires, the batch stops and reports where it stopped. The returned evidence helps the model decide whether to inspect, retry, or change approach. It does not turn an unverified input into a verified outcome.
Sam: Give me a case where combining channels mattered, rather than merely offering several ways to click.
Alex: In a recorded GIMP task, the agent entered “Square” as a layer name through accessibility controls. The accessibility feedback did not expose the updated field value. A screenshot confirmed the text visually, and the agent then resumed accessibility-based actions to accept the dialog.
Sam: That separates observation from execution. The best way to see a result need not be the best way to perform the next action.
Alex: The trajectories support that pattern. Several models used screenshots to observe while executing through accessibility controls. HCU adds no separate planner or team of subagents. Channel selection stays with the existing model inside its usual action loop.
Sam: Switching channels also sounds risky. A dialog could disappear while the model still holds coordinates from an old screenshot.
Alex: That is what the action guards address. Before an accessibility action, HCU resolves the reference against the live element and checks its identity. Before coordinate input, it checks whether the relevant window layout still matches the screenshot. A detected mismatch blocks input and supplies fresh evidence.
Sam: So these are checks on targets and layouts, not a guarantee that the model’s whole plan is correct.
Alex: Yes. Their scope is narrower than task correctness. They help prevent acting on stale evidence, but the model still has to choose the right operation and verify the result.
Sam: Let us turn to the evidence. Which comparison most cleanly isolates the interface?
Alex: The main matched comparison uses GPT-6 Astra on the full OSWorld desktop-workflow benchmark, covering 108 tasks. HCU and CUA Driver use the same surrounding agent software and programmatic tools. HCU reaches about 72 percent average task score, against about 69 percent for CUA Driver.
Sam: Is that score a completion rate, or can an unfinished task still earn credit?
Alex: It can earn partial credit. That headline score averages progress across tasks; it does not mean nearly all those tasks were completed. The paper also reports full-credit rates, which are much lower. In this matched comparison, the full-credit improvement is smaller than the average-score improvement.
Sam: And did the higher score require more interaction or more spending?
Alex: HCU used about 30 percent fewer tool calls than CUA Driver. It also took less time, while monetary cost was almost unchanged. That matters: fewer calls do not automatically translate into proportionally cheaper model use.
Sam: A single model could happen to like this tool format. Does the advantage survive changing models?
Alex: In a separate comparison on the earlier OSWorld benchmark, every evaluated model improved its score and lowered cost. But they used the interface differently. Most favored accessibility calls for desktop interaction; Claude favored visual tools. Its overall elapsed time increased slightly, despite its better score.
Sam: That makes the flexible interface sound more defensible than prescribing one ideal workflow. What did removing components reveal?
Alex: Accessibility contributed the largest score gain and most resource savings in the component tests. Removing visual tools left scores within run-to-run variation, but increased interaction steps. The supplementary usage guide also mattered: malformed-call rejections fell from about seventeen percent to about one percent with the guide.
Sam: Then the result belongs partly to the instructions, not just the software. What else should keep us from overgeneralizing?
Alex: The strongest claim concerns the package as tested. Accessibility coverage depends on what applications expose; custom-drawn content may remain inaccessible. Additional tests support transfer across desktop platforms, but the macOS comparison uses a selected, small subset requiring graphical interaction. Also, those additional tests use a refined release, not exactly the main experimental version.
Sam: Given those qualifications, who should read the full document, and where should they start?
Alex: Agent builders should start with Section Three, on the interface, then the component ablations in Section Four. Those separate the contributions of accessibility, visual tools, and instructions. Before reproducing anything, check the evaluated-versus-released distinction in Appendix B. Researchers studying agent behavior should read the cross-model analysis, especially how observation and execution split across channels.
Sam: And what should everyone else carry away?
Alex: Judge an agent by the routes you let it take, not only by the model inside it.
Sam: Thanks for listening.