Are We There Yet? Assessing Computer-Use Agents for Blind Users’ Accessible Interaction with Desktop Applications

This study investigates the real-world effectiveness of computer-use agents (CUAs) for blind screen-reader users. The researchers developed OLLA, an accessibility-focused interaction layer that allows blind users to issue natural-language commands to an agent while receiving non-visual feedback. Over a three-week longitudinal diary study, 8 blind participants issued 1,258 commands across 12 common desktop applications. The researchers evaluated GPT-5 during the study and performed a controlled re-execution of the same commands using four other models (Claude Sonnet, Gemini 2.5 CU, UI-TARS, and Qwen3-VL) to compare performance and identify systematic failure modes.

Performance across all models was modest, with GPT-5 achieving the highest success rate at 52.5%. While agents could handle simple, direct tasks like changing font settings or opening menus, they struggled significantly with complex, multi-step workflows. Trace analysis revealed that failures were not random but clustered around specific bottlenecks: agents often hallucinated controls that did not exist (grounding errors), failed to navigate through nested menus (hidden path discovery), or lost track of specific user requirements (constraint-binding failures) after making initial progress. Furthermore, agents often relied on memorized, generic procedures rather than adapting to the specific interface state currently visible on the user's screen.

Interviews with participants revealed that blind users do not necessarily want fully autonomous agents. Instead, they envision CUAs as collaborative assistants that can help them understand complex interfaces, troubleshoot errors, and learn new application workflows. Participants expressed a strong desire for "human-in-the-loop" interaction, where the agent explains its plan, asks for clarification, and confirms critical actions before execution, particularly when dealing with sensitive data or irreversible changes.