Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents

Computer Use Agents (CUAs) are increasingly capable of performing complex, multi-step tasks within graphical user interfaces (GUIs). However, these agents often operate over long durations, requiring users to manage multiple tasks simultaneously. Current interfaces for these agents rely heavily on text-based logs, which force users to either constantly monitor the agent to ensure progress or risk losing track of the agent's state, leading to inefficient context switching and delayed error detection.

The authors developed Sidekick, a prototype designed to bridge these communication gaps across three interaction stages:

1. Background Execution: Sidekick uses an always-on-top peripheral display that provides concise text summaries, visual thumbnails, and color-coded status cues (e.g., green for progress, red for errors) to allow for unobtrusive monitoring.
2. Context Resumption: When a user returns to the agent, Sidekick generates a multimodal summary—combining speech and visual replays of completed actions—to help the user quickly understand what the agent achieved while they were away.
3. Foreground Interaction: When the user is actively monitoring the agent, Sidekick provides real-time transparency by verbalizing and visualizing the agent's internal reasoning and specific GUI actions (e.g., clicks and typing).

A mixed-methods study with 30 participants compared Sidekick against baseline systems that used standard chat or peripheral text-only displays. Participants performed a primary arithmetic task while the CUA handled a secondary spreadsheet-filling task. The results showed that Sidekick significantly improved multitasking performance. Participants were better able to maintain awareness of the agent's progress and intervene when errors occurred, without the high cognitive cost associated with manually parsing dense text logs.