Instruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-Distillation

Reinforcement Learning (RL) post-training often suffers from support shrinkage, where the model's output distribution narrows and loses the ability to explore diverse, correct reasoning paths. This paper investigates how to broaden the range of behaviors (support expansion) during training to improve the performance of smaller Large Language Models (LLMs) on complex mathematical reasoning tasks.

The authors introduce Instruction-Conditioned Exploration (ICE), which supplements standard task prompts with diverse, high-level behavioral instructions during training. This forces the model to generate a wider variety of reasoning strategies. To ensure these explored behaviors are effectively transferred to the final model, the authors propose Asymmetric-RL/SD. This training objective uses a shared-parameter architecture where a teacher policy is conditioned on instructions and optimized via RL, while a student policy (the unconditioned model used at test time) is updated via self-distillation to mimic the teacher's successful, correctness-filtered trajectories.

Empirical testing on the Qwen3-1.7B model demonstrates that ICE with Asymmetric-RL/SD consistently outperforms the baseline DAPO method on mathematical reasoning benchmarks. At a 4K response length, the method achieved a 5.0% relative improvement in pass@1 performance. The authors also found that the performance gains persist when extending the context length to 8K. However, the benefits were not observed when scaling the model to 4B parameters, suggesting that the effectiveness of this exploration strategy may be sensitive to model capacity.

As smaller LLMs become increasingly viable for deployment on consumer hardware, improving their reasoning capabilities without requiring massive scale is critical. This work provides a practical, instruction-based framework for enhancing exploration in RL post-training, offering a way to mitigate the common problem of entropy collapse and distribution sharpening in smaller models.