Where Do CoT Training Gains Land in LLM based Agents?

Chain-of-thought (CoT) reasoning is a standard technique for improving language model agents, yet it remains unclear whether CoT training actually improves the model's ability to revise actions through reasoning or simply makes the model better at predicting actions directly from the prompt. This paper investigates the source of performance gains in CoT-trained agents by comparing the quality of actions generated with and without CoT.

The authors define two decoding modes: 'prompt actions' (predicted directly from the prompt) and 'CoT actions' (predicted after generating a reasoning trace). By tracking these across training checkpoints in environments like ALFWorld, ScienceWorld, and BFCL, the researchers analyze whether the performance gap between these two modes widens. They further use 'conflicting-trace tests'—where a reasoning trace is replaced with one implying a different action—to measure whether the model remains anchored to the prompt or follows the reasoning.

The study finds that prompt-action quality and CoT-action quality improve in parallel throughout training, with the relative advantage of CoT remaining largely flat. This indicates that training does not disproportionately enhance the reasoning-based revision process. Instead, the model becomes increasingly reliant on the prompt to determine its final action. Mechanistic analysis reveals that because prompts are typically much longer than reasoning traces, they receive the majority of attention and gradient mass during training, creating a structural optimization bias that favors direct prompt-to-action shortcuts. Based on this, the authors demonstrate that selectively masking action-token supervision during training improves out-of-domain generalization.

These findings suggest that current CoT training regimes may be inadvertently reinforcing shortcut learning rather than genuine reasoning. For researchers, this implies that simply increasing CoT training data may not lead to more 'thoughtful' agents, but rather to models that are more efficient at pattern-matching from the prompt. The proposed intervention—reducing action-token supervision—offers a practical way to encourage models to rely more on their generated reasoning traces.