VisCritic: Visual State Comparison as Process Reward for GUI Agents

GUI agents often struggle with long-horizon tasks because they lack reliable, step-level verification mechanisms. Existing methods typically rely on textual reasoning or structured checklists to verify actions, which fails to capture the inherently visual nature of GUI interactions—such as buttons changing color, new dialogs appearing, or pages scrolling. This paper asks: can we improve GUI agent reliability by directly comparing visual state changes as a process reward signal?

The authors introduce VisCritic, a plug-and-play framework that functions as an inference-time module for GUI agents. It consists of three primary components:

• Visual Difference Encoder (VDE): A Siamese vision transformer that extracts semantic differences between consecutive screenshots, computing a "change magnitude map" to identify meaningful UI updates.
• Action-Aware Critic Head: A multi-task head that fuses these visual difference features with the agent's action and task instruction to predict action success, task progress, and specific error types.
• Weakly Supervised Training: A pipeline that generates training data from existing trajectory datasets without requiring additional human labels, using perturbation strategies like action/state mismatches and no-op detection.

Findings

VisCritic was evaluated across five benchmarks, including web, mobile, and desktop environments. The results demonstrate that it consistently improves the performance of diverse base agents (such as SeeClick, ShowUI, and Qwen2.5-VL) by providing a more reliable verification signal than text-based baselines. The framework is particularly effective in web-based tasks where error accumulation is a major bottleneck. Additionally, the model provides qualitative diagnostic cues through attention maps that highlight the specific regions of the screen where the agent's action caused (or failed to cause) a change.