Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It

This paper investigates why reinforcement learning (RL) for multi-step tool-use tasks often leads to training instability and catastrophic performance collapse. The authors seek to understand the underlying mechanism of these failures and determine how various supervisory signals can be integrated into the RL process to maintain stable, high-quality agentic behavior.

The researchers analyze the training dynamics of LLMs (specifically Qwen2.5-1.5B-Instruct and Qwen3-1.7B) on the BFCL-V3 benchmark. They identify that RL-based optimization often causes a shift in probability mass toward specific control tokens, leading to malformed tool-invocation structures. To address this, they propose a unified framework to test five different supervisory signals: SFT-then-RL, off-policy supervision, hint-based guidance, erroneous trajectory supervision, and a novel method called Process Reflection Supervision. These methods are tested under both synchronous and interleaved training paradigms to compare their impact on stability and generalization.

The study reveals that RL failure in tool-use tasks is primarily a structural collapse problem rather than a loss of reasoning capability. The model's underlying competence remains intact, but it becomes obscured by degenerate control-token patterns. The authors find that simply mixing supervision with RL is insufficient; synchronous methods often suffer from distribution mismatch. Conversely, interleaving supervised fine-tuning (SFT) with RL significantly improves stability. Furthermore, the newly introduced Process Reflection Supervision—which converts intermediate reasoning steps into explicit textual guidance—proves effective in regulating token-level execution patterns, enabling more robust training.

As LLMs are increasingly deployed as autonomous agents, the ability to reliably train them for multi-step tool use is critical. This paper provides a mechanistic explanation for why RL often fails in these settings, shifting the focus from "model capability" to "structural token dynamics." By offering a systematic taxonomy of supervisory signals, the work provides practitioners with actionable strategies to stabilize agentic training without sacrificing the performance gains typically associated with RL.