GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Models

Large Vision-Language Models (LVLMs) often utilize Mixture-of-Experts (MoE) architectures to scale parameter capacity without linearly increasing inference costs. Traditional MoE models typically employ a static Top-k routing strategy, which assigns a fixed number of experts to every input token. This approach is inherently inefficient, as it fails to account for the varying complexity of input data, leading to wasted computational resources on simple tokens and insufficient capacity for complex ones. This paper investigates how to implement a dynamic routing mechanism that balances model complexity with performance.

The authors frame token routing as an information encoding task and propose viewing dynamic routing through the lens of Minimum Description Length (MDL). The core idea is that the optimal routing strategy should minimize the total length required to describe both the model and its fit to the data. Since calculating MDL directly is computationally prohibitive, the authors identify gating entropy—a measure of routing uncertainty—as a reliable proxy.

They introduce GeMoE (Gating Entropy-based Uncertainty-aware Adaptive Routing), which uses an Expert Assignment Predictor (EAP) to map a token's gating entropy to the number of experts it requires. To ensure this mapping is effective, they implement a monotonic loss function during training, which forces the model to assign more experts to tokens with higher gating entropy, where the potential information gain is greatest.

GeMoE demonstrates that gating entropy is a highly effective indicator for adaptive resource allocation. By dynamically adjusting the number of experts per token, the model significantly improves expert activation sparsity—reducing the average number of active experts by 36.5%—while maintaining 99.5% of the performance achieved by static Top-k routing. The authors validate this across multiple backbones, showing that the method successfully balances computational efficiency with task-specific performance.