ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning

On-policy training for LLMs typically relies on successful expert trajectories to guide learning. When expert models fail on difficult problems, these trajectories are usually discarded as negative samples. This paper investigates whether these discarded 'Golden Negative Trajectories' (GNTs) contain latent reasoning signals that can be leveraged to improve model performance.

The authors identify a 'Reflection Advantage,' where models tasked with reflecting on and correcting a flawed expert trajectory perform better than those attempting to solve the problem from scratch. Based on this, they propose ReflectRL, a plug-and-play framework that integrates GNTs into existing on-policy training methods like Reinforcement Learning with Verifiable Rewards (RLVR) and On-Policy Distillation (OPD). ReflectRL uses a 'Reflective-to-Direct Policy Transition' mechanism, which gradually shifts the model from training on reflective prompts (which include the GNT) to direct prompts (which do not), ensuring the model internalizes the error-correction behavior for inference.

ReflectRL consistently improves reasoning performance across 9 benchmarks, 4 LLM backbones, and 4 training methods. The authors demonstrate that GNTs are uniquely beneficial because they maintain higher-quality reasoning prefixes before localized errors occur, providing a structured 'map' of where reasoning typically goes wrong. The transition mechanism effectively transfers these learned error-avoidance capabilities to the direct reasoning interface, resulting in higher accuracy and more concise reasoning paths without increasing computational overhead during inference.