UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models

As large-scale recommendation models increasingly adopt Mixture-of-Experts (MoE) architectures to handle complex user behavior, they face a critical deployment challenge: trained checkpoints often contain more experts than are necessary for efficient serving. The authors investigate how to compress these trained MoE models into smaller, standard MoE architectures under a fixed expert budget without introducing complex online modules that would complicate inference.

UniMoMo treats the expert-bank reduction as a constrained graph coarsening problem. Instead of relying on parameter distance, which fails to capture how experts behave on specific recommendation tasks, the framework uses an unlabeled calibration set to evaluate functional similarity. It constructs an expert-affinity graph based on how experts respond to shared recommendation states.

To ensure performance stability, the authors introduce a layer-adaptive protection mechanism. This mechanism uses routing entropy to identify high-traffic experts—those that significantly influence the model's output—and restricts them from being repeatedly merged. Finally, the framework reconstructs the new, merged experts using a traffic-weighted parameter fusion, followed by a least-squares correction to account for the non-linearities inherent in SwiGLU blocks. The resulting model is then briefly fine-tuned to adapt to the new architecture.

UniMoMo demonstrates that it is possible to significantly reduce the expert bank of a trained recommendation model while maintaining high ranking utility. Across three datasets (Amazon Beauty, KuaiRec, and TenRec) and varying MoE depths, the four-expert compressed checkpoints achieved 99.92%–102.30% of the original model's NDCG@10 performance while providing 1.28×–1.63× speedups on A100 GPUs. Even more aggressive compression to a two-expert configuration yielded 98.36%–104.24% of the original performance with speedups reaching up to 2.21×.