Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold

Standard reinforcement learning (RL) often produces models that are strong but incomplete, as they tend to amplify only a subset of valid reasoning strategies. The authors investigate whether reasoning performance can be improved by treating RL-trained policies as local probes of a multi-basin solution manifold rather than globally reliable supervisors, and whether these diverse strategies can be effectively compressed into a single, superior student model.

The authors propose an 'expand-then-compress' framework. In the expansion stage, they use Residual Group Relative Policy Optimization (RGRPO) to train a sequence of teachers from a common initialization. Each subsequent teacher is directed toward examples not yet reliably covered by the accumulated union of previous teachers, forcing them to specialize in different reasoning basins. In the compression stage, they use Reliability-Gated Teacher-Union On-policy Distillation (TU-OPD) to distill these specialists into a single student. This process includes a Consensus-Residual Decomposition, which ensures that the student learns both the shared behaviors of the teacher union and the unique, specialist token preferences of the best-performing teacher for any given example.

Experiments across mathematical reasoning, code generation, and instruction following demonstrate that the resulting student model consistently outperforms the strongest individual teacher in the union. By deliberately constructing and compressing a complementary set of teachers, the authors achieve relative improvements of 2.0% in math, 8.3% in code generation, and 6.9% in instruction following, while maintaining the efficiency of single-model inference. The results confirm that stronger reasoning capabilities can be obtained by aggregating diverse, complementary solution modes rather than simply selecting a single 'best' teacher.