Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

Large language models are often adapted to specialized domains using Low-Rank Adaptation (LoRA). However, composing multiple independently trained LoRA adapters into a single model is challenging because standard Mixture-of-Experts (MoE) soft-routing techniques blend expert updates, which violates the unit-scale assumption under which individual LoRA modules were originally trained. This paper introduces Hard-Routed MoR-LoRA, a two-stage framework designed to integrate frozen reasoning experts without retraining them. In the first stage, domain-specific LoRA adapters are trained using Reinforcement Learning from Verifiable Feedback (RLVF) to acquire reasoning capabilities. In the second stage, these experts are frozen, and a lightweight router—trained via a straight-through estimator (STE)—is used to perform hard, token-level expert selection.

Experiments across five benchmarks and multiple model scales demonstrate that RLVF produces significantly stronger reasoning experts compared to supervised fine-tuning, particularly for larger models. When integrating these experts, the proposed hard-routing approach consistently matches or exceeds the performance of soft-routing baselines. Crucially, hard routing achieves this while keeping all pretrained experts frozen and requiring substantially fewer trainable parameters. The authors further observe that normalized soft-routing mixtures often concentrate most of their probability mass on a single expert, suggesting that hard unit-scale routing is a more direct and efficient abstraction for expert composition.

This work provides a practical solution for modular LLM adaptation in environments where original training data cannot be shared due to privacy or regulatory constraints. By decoupling reasoning acquisition from expert selection, the framework allows researchers to build a library of specialized experts and compose them dynamically. This modularity reduces the need for expensive full-model fine-tuning and avoids the parameter overhead associated with retraining experts during integration.