Bonan Zhang, Shiyu Dong, Quan Hung Tran, Katharina Gschwind, Shuqi Yang, Sijia Chen, Adel Ahmadyan, Seungwhan Moon, Lu Zhang, Ahmed Kirmani, Babak Damavandi, Anuj Kumar
5 min
Vision encoders are fundamental components of vision-language models, but scaling them densely increases computational cost and inference latency. While Mixture-of-Experts (MoE) has successfully enabled efficient scaling in large language models, its design for CLIP-style vision encoders has remained underexplored at state-of-the-art levels. This paper presents MoE-ViE, a systematic study of MoE architecture scaling for vision encoders. Rather than using conventional MoE as a simple drop-in replacement for dense Multi-Layer Perceptrons, the authors adopt a fine-grained expert topology with reduced hidden widths, combined with shared experts and a Sigmoid-based gating mechanism to avoid expert competition.
To make sparse scaling practical and effective for both images and videos, the authors introduce several key innovations. First, they propose an auxiliary-loss-free balancing variant that uses z-score normalization on token load deviations to maintain stable expert utilization without training objective interference. Second, to overcome memory-bandwidth bottlenecks and hardware idle times typical of sparse operations, they develop a specialized Triton kernel utilizing Grouped GEMM and kernel fusion, achieving over 2.5x speedups in inference latency. Finally, for video understanding, they combine frame-level distillation with freezing mechanisms for MoE experts to adapt the model to video data without degrading pretrained image capabilities.
Pretrained on large-scale image-text pairs and finetuned on video datasets, MoE-ViE models consistently outperform dense counterparts under matching compute budgets across all evaluated scales. On zero-shot image classification, retrieval, and fine-grained benchmarks, MoE-ViE matches or exceeds state-of-the-art dense encoders. Notably, the largest MoE-ViE model matches the performance of a dense encoder 1.7x its size while operating at only 76% of its latency, and surpasses it on several challenging benchmarks when aligned with a language model.
Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
Sam: Okay, so that handles the image side. What about video? I'd have thought a model good at images would naturally extend to video.
Alex: You'd think so, but it's more complicated. Video asks the model to handle something new — motion, changes across frames, relationships between moments in time. When you train on video, that pressure can pull the model away from the image patterns it already learned. It's a bit like a musician who's great at classical piano, then spends months learning jazz. They can start to lose the precision they had before.
Sam: So how do you stop that drift?
Alex: Two ways. First, the paper keeps the original image-trained model running alongside the new one during video training, and compares their outputs. If the video model's frame-by-frame understanding starts drifting away from what the image model would have said, that acts as a corrective signal. Second, some of the specialist parts get frozen — their weights don't change during video training — so the knowledge stored in those parts stays intact.
Sam: So it's less about building a video model from scratch and more about carefully extending an image model without breaking what it already knows.
Alex: That's a good way to put it. The paper suggests that combining both strategies — the guidance signal and the freezing — preserves image ability much better than video training alone.
Sam: And what about actually running this efficiently? A sparse model with lots of routing logic could easily end up slower than just using a bigger dense model.
Alex: That's where the implementation work comes in. The standard way to handle sparse routing on a GPU involves a lot of back-and-forth data movement, which creates a bottleneck. The authors rewrote that step using what are called fused operations — instead of doing things one at a time and shuffling data around between each step, they bundle the work together so the GPU handles it in fewer, more efficient passes.
Sam: And that made a meaningful difference in speed?
Alex: It did. With that change, the model ran at roughly three-quarters the latency of a stronger dense baseline, while maintaining similar accuracy. So you're getting comparable quality at noticeably lower cost.
Sam: So the paper isn't just proposing an architecture — it's also showing that the architecture can actually run in practice.
Alex: Right. That's an important part of the contribution. A design that's theoretically efficient but slow in practice doesn't help anyone. The paper is careful to show that the sparse approach works at the implementation level, not just on paper.
Sam: What's the takeaway, then? Is this a solved problem, or more of a step in a direction?
Alex: The authors are measured about it. The paper shows that sparse vision encoders can scale, stay practical, and retain image understanding when extended to video. It's a careful step, not a final answer. But it does point to a useful direction — that the right architecture choices, combined with thoughtful training strategies, can make vision models more capable without simply making them more expensive to run.
Sam: That's a more useful result than it might sound at first.
Alex: It is. The question of how to scale vision models efficiently matters a lot as these systems get used in more demanding settings. Thanks for listening to ResearchPod.