SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment

Sparse Mixture-of-Experts (MoE) models often struggle with low-resource languages because their routing networks fail to activate the same specialized experts that are triggered by high-resource inputs. This leads to cross-lingual routing divergence, where semantically identical inputs across different languages result in disparate expert activation pathways. The authors investigate whether explicitly aligning these internal routing distributions can improve performance in low-resource settings.

The authors propose Semantically Anchored Routing Alignment (SARA), a three-stage framework designed to transfer routing logic from high-resource languages to low-resource ones. First, they construct a parallel instruction dataset by translating verified high-resource interactions into target low-resource languages. Second, they perform forward propagation on the high-resource data to extract stable 'routing priors'—the probability distributions of expert activation. Finally, they fine-tune the model using a composite loss function that includes a symmetric Jensen-Shannon (JS) divergence constraint. This constraint penalizes deviations between the routing distributions of the low-resource inputs and the high-resource anchors, effectively forcing the model to adopt consistent expert-selection logic regardless of the input language.

SARA consistently outperforms standard instruction tuning and existing alignment methods across multiple benchmarks, including Global-MMLU, BELEBELE, and MGSM. For instance, the framework achieved performance gains of 0.8% on Qwen3-30B-A3B and 1.2% on Phi-3.5-MoE-instruct on the Global-MMLU benchmark. The results demonstrate that aligning internal routing mechanisms is more effective than traditional output-logit distillation or hidden-state alignment, as it directly addresses the mechanistic bottleneck of expert selection in sparse architectures.