Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination.
Alex: Making an agent use a computer like a person can hold it back. This study finds that agents do better when they can combine code, named controls, and screenshots. The gain is not just fewer clicks. On desktop benchmarks, the authors report better task scores with less interaction.
Sam: That sounds like changing the tools rather than making the model smarter. Whose work is this, and what are they proposing?
Alex: It is a method paper and preprint by Haoqing Wang and colleagues, primarily at Microsoft Research. Their interface is called Hybrid Computer Use. It lets an existing agent choose how to operate the computer, without retraining its model.
Sam: So for researchers building agents, the interface itself becomes something to test, not just plumbing around the model.
Alex: That is the practical stake. We will look at what the interface adds, how they tested it, and why better average scores still leave substantial work unfinished.
Sam: Start with the constraint. Why would copying human computer use be inefficient for a model?
Alex: A screenshot-based agent repeatedly inspects an image, locates a control, and sends mouse or keyboard input. But an application may already expose that control’s name and value through accessibility software. Or the task may involve data that code could process directly, without dragging anything around.
Sam: Those alternatives already existed. Is the contribution simply giving the agent access to all of them?
Alex: No. The authors explicitly describe earlier hybrid systems. Their main comparison, CUA Driver, already supports programmatic, accessibility, and visual interaction. The contribution is how HCU organizes those capabilities: compact observations, batches of actions, built-in feedback, and checks against outdated desktop state.
Sam: Then let us unpack the accessibility part. What does the agent actually receive instead of pixels?
Alex: An accessibility tree: structured information an application exposes about its controls. That includes roles, names, values, and states, such as whether a checkbox is checked. HCU gives controls short references, so the model can address a particular button rather than estimate its screen coordinates.
Sam: Application trees can be enormous. Does replacing an image with a giant text dump really save effort?
Alex: HCU keeps the acquired records internally but shows the model a compact view. It removes empty structural wrappers and condenses repeated content. References are assigned before that compression, so shortening the display does not change which element a reference identifies. A targeted search can retrieve controls left out of the compact view.
Sam: So the model gets a readable working view, while the interface retains the detail needed to act. What reduces the repeated back-and-forth?
Alex: The agent can combine known actions into a batch. It can also insert a wait for an expected state before continuing. The same call returns observed changes, called an AX diff, rather than requiring a separate inspection every time.
Sam: A successful tool call does not necessarily mean the application did what you wanted. How does the interface handle that distinction?
Alex: It separates input delivery from evidence of an effect. If an action fails or a wait expires, the batch stops and reports where it stopped. The returned evidence helps the model decide whether to inspect, retry, or change approach. It does not turn an unverified input into a verified outcome.
Sam: Give me a case where combining channels mattered, rather than merely offering several ways to click.
Alex: In a recorded GIMP task, the agent entered “Square” as a layer name through accessibility controls. The accessibility feedback did not expose the updated field value. A screenshot confirmed the text visually, and the agent then resumed accessibility-based actions to accept the dialog.
Sam: That separates observation from execution. The best way to see a result need not be the best way to perform the next action.
Alex: The trajectories support that pattern. Several models used screenshots to observe while executing through accessibility controls. HCU adds no separate planner or team of subagents. Channel selection stays with the existing model inside its usual action loop.
Sam: Switching channels also sounds risky. A dialog could disappear while the model still holds coordinates from an old screenshot.
Alex: That is what the action guards address. Before an accessibility action, HCU resolves the reference against the live element and checks its identity. Before coordinate input, it checks whether the relevant window layout still matches the screenshot. A detected mismatch blocks input and supplies fresh evidence.
Sam: So these are checks on targets and layouts, not a guarantee that the model’s whole plan is correct.
Alex: Yes. Their scope is narrower than task correctness. They help prevent acting on stale evidence, but the model still has to choose the right operation and verify the result.
Sam: Let us turn to the evidence. Which comparison most cleanly isolates the interface?
Alex: The main matched comparison uses GPT-6 Astra on the full OSWorld desktop-workflow benchmark, covering 108 tasks. HCU and CUA Driver use the same surrounding agent software and programmatic tools. HCU reaches about 72 percent average task score, against about 69 percent for CUA Driver.
Sam: Is that score a completion rate, or can an unfinished task still earn credit?
Alex: It can earn partial credit. That headline score averages progress across tasks; it does not mean nearly all those tasks were completed. The paper also reports full-credit rates, which are much lower. In this matched comparison, the full-credit improvement is smaller than the average-score improvement.
Sam: And did the higher score require more interaction or more spending?
Alex: HCU used about 30 percent fewer tool calls than CUA Driver. It also took less time, while monetary cost was almost unchanged. That matters: fewer calls do not automatically translate into proportionally cheaper model use.
Sam: A single model could happen to like this tool format. Does the advantage survive changing models?
Alex: In a separate comparison on the earlier OSWorld benchmark, every evaluated model improved its score and lowered cost. But they used the interface differently. Most favored accessibility calls for desktop interaction; Claude favored visual tools. Its overall elapsed time increased slightly, despite its better score.
Sam: That makes the flexible interface sound more defensible than prescribing one ideal workflow. What did removing components reveal?
Alex: Accessibility contributed the largest score gain and most resource savings in the component tests. Removing visual tools left scores within run-to-run variation, but increased interaction steps. The supplementary usage guide also mattered: malformed-call rejections fell from about seventeen percent to about one percent with the guide.
Sam: Then the result belongs partly to the instructions, not just the software. What else should keep us from overgeneralizing?
Alex: The strongest claim concerns the package as tested. Accessibility coverage depends on what applications expose; custom-drawn content may remain inaccessible. Additional tests support transfer across desktop platforms, but the macOS comparison uses a selected, small subset requiring graphical interaction. Also, those additional tests use a refined release, not exactly the main experimental version.
Sam: Given those qualifications, who should read the full document, and where should they start?
Alex: Agent builders should start with Section Three, on the interface, then the component ablations in Section Four. Those separate the contributions of accessibility, visual tools, and instructions. Before reproducing anything, check the evaluated-versus-released distinction in Appendix B. Researchers studying agent behavior should read the cross-model analysis, especially how observation and execution split across channels.
Sam: And what should everyone else carry away?
Alex: Judge an agent by the routes you let it take, not only by the model inside it.
Sam: Thanks for listening.