A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

The Question of Post-Training Gains

Reinforcement learning (RL) with verifiable rewards, particularly GRPO, is widely used to refine language and vision-language models. While this approach has shown success in reasoning and coding, it remains unclear whether it adds genuine new skill to web agents that have already reached a high level of competence through supervised fine-tuning (SFT). This study investigates whether GRPO adds capability or merely reshapes existing behavior in a controlled environment.

Controlled Null and Positive Control

The researchers conducted a rigorous grid search across 18 configurations—varying learning rates, KL weights, seeds, and initialization—on the MiniWoB benchmark. The results reveal a consistent null: no configuration credibly improves the success rate of the supervised baseline on tasks the agent has already mastered. In fact, moderate to high learning rates actively degrade or collapse the agent's performance.

To ensure this null result was not due to a broken training pipeline, the researchers performed a positive control. By applying the exact same GRPO recipe to tasks where the model's sampled policy already outperformed its greedy policy (i.e., tasks with 'headroom'), the agent achieved a 22-point gain. This demonstrates that GRPO is not inherently broken, but rather that its efficacy is strictly gated by the existence of a superior policy within the model's current sampling distribution.

Mechanistic Failure Analysis

The study further analyzed why higher learning rates cause the agent to fail. The researchers identified a double dissociation: moderate learning rates degrade the agent by destroying the effective rank of late-layer attention and MLP blocks, while high learning rates cause a total collapse of the model's output. Through causal weight grafting—where specific components were reset to their initial states—the team proved that the degradation is localized to these compute blocks. Interestingly, while the embedding weights showed the most movement during training, they were found to be causally inert regarding the performance failure, serving as a 'red herring' in the analysis.