TACTILE: GIVING COMPUTER-USING AGENTS HANDS AND FEET

Computer-using agents often struggle with desktop tasks because they rely on brittle 'screenshot-first' control. By treating the screen as a collection of pixels, agents must guess coordinates and hope for the intended state change, which conflates perception, action, and verification. Tactile introduces a reusable tool layer that provides agents with a more reliable 'hands and feet' by exposing software actions as semantic, verifiable, and auditable objects.

Tactile organizes desktop interaction into an 'observe-ground-act-verify' loop. Instead of forcing the agent to predict raw mouse coordinates, Tactile compiles heterogeneous UI evidence—such as operating-system accessibility trees, OCR-grounded text, and visual fallback regions—into a compact, ranked set of target candidates. This allows the agent to select a semantic target (e.g., an 'Enabled Save Button') and lets the runtime determine the safest way to interact with it, such as using a native accessibility press action versus a simulated mouse click.

Tactile does not discard visual information; instead, it uses an 'accessibility-first' ladder to prioritize the most reliable signal available. It prefers native semantic actions when an application exposes them, falls back to OCR-grounded coordinates when text is the best locator, and uses visual fallback for semantically opaque regions like custom canvases or remote desktops. This approach ensures that the agent always uses the richest available structure while maintaining the ability to operate in any graphical environment.

Evaluations on macOSWorld-style tasks demonstrate that Tactile consistently improves success rates across multiple agents, including Codex, Claude Code, OpenCode, and Goose. By providing a standardized interface for desktop operation, Tactile allows developers to debug failures more effectively, as the system retains full provenance for every action, making it clear whether a failure occurred during perception, target selection, execution, or verification.