GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

This paper investigates the fundamental performance differences between GUI-based and CLI-based computer-use agents. While both modalities aim to automate desktop tasks, they rely on vastly different interaction paradigms: GUI agents operate through visual screen feedback and mouse/keyboard inputs, whereas CLI agents utilize programmatic skill layers to manipulate application state. Previous research has struggled to isolate the impact of these modalities because benchmarks often conflate tasks, initial states, and action spaces. This study introduces a controlled, matched execution-layer benchmark of 440 tasks across 18 applications to provide a fair comparison.

The researchers developed a benchmark where both GUI and CLI agents receive identical goals, initial states, and final-state verifiers. By restricting agents to their respective modality-native actions—screen-only operations for GUI and curated skill-based commands for CLI—the authors isolated the effect of the interaction interface itself. They further diagnosed the CLI performance gap by testing a 'patched-skill' setting, where missing skill paths were repaired based on verifier feedback to determine if the limitation was due to the model or the interface.

The study reveals that the strongest GUI agent (59.1% pass rate) outperforms the strongest original-skill CLI agent (48.2%). However, when CLI agents are provided with augmented, verifier-guided skills, their success rate jumps to 69.3%, indicating that the CLI deficit is largely a result of incomplete skill coverage. The authors identify distinct failure modes: GUI agents struggle with visual grounding and long-horizon workflow execution, while CLI agents are hindered by missing skill operations, the need to reconstruct implicit application defaults, and unobservable application semantics.