Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?

As computer-use agents (CUAs) become more prevalent, they are typically evaluated on end-to-end task success. However, this high-level metric often masks intermediate failures in visual reasoning, such as the inability to distinguish between relevant state changes and stale or unrelated GUI noise. This paper asks: can current vision-language models (VLMs) accurately interpret the causal transitions between desktop GUI states, and can they reject irrelevant visual information that might poison their planning context?

The authors introduce Desktop-Delta Bench (DDB), an offline diagnostic benchmark consisting of 2,013 human-verified instances derived from Linux desktop workflows. DDB evaluates models through two primary tasks:

1. Temporal Ordering: Models must reconstruct the correct chronological sequence of three screenshots. In some instances, one screenshot is a 'decoy' from a different trajectory, testing the model's ability to reject irrelevant context.
2. Single-Action Reconstruction: Given a pair of screenshots (before and after an action), models must identify the action family (e.g., click, drag, scroll) and its associated parameters (e.g., coordinates, text input).

These tasks are designed to probe three specific failure dimensions: state verification (detecting hidden or persistent changes), source tracking (maintaining consistency across applications), and context-aware control (handling ambiguity and recovery).

Evaluation of eight frontier model families shows that desktop transition understanding remains far from saturated. In temporal ordering, models achieved an exact-match rate of approximately 65%, with little improvement when provided with task context. Error analysis indicates that models often default to simply copying the presented order of images rather than reasoning through the visual evidence. In single-action reconstruction, models were significantly better at locating UI elements than identifying the correct action family, with 'drag' actions proving particularly difficult to classify compared to simple clicks. These results suggest that while models can ground UI elements, they lack the robust causal understanding required for reliable, long-horizon desktop automation.