Bonan Zhang, Shiyu Dong, Quan Hung Tran, Katharina Gschwind, Shuqi Yang, Sijia Chen, Adel Ahmadyan, Seungwhan Moon, Lu Zhang, Ahmed Kirmani, Babak Damavandi, Anuj Kumar
5 min
Abstract
Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
Sam: Okay, so that handles the image side. What about video? I'd have thought a model good at images would naturally extend to video.
Alex: You'd think so, but it's more complicated. Video asks the model to handle something new — motion, changes across frames, relationships between moments in time. When you train on video, that pressure can pull the model away from the image patterns it already learned. It's a bit like a musician who's great at classical piano, then spends months learning jazz. They can start to lose the precision they had before.
Sam: So how do you stop that drift?
Alex: Two ways. First, the paper keeps the original image-trained model running alongside the new one during video training, and compares their outputs. If the video model's frame-by-frame understanding starts drifting away from what the image model would have said, that acts as a corrective signal. Second, some of the specialist parts get frozen — their weights don't change during video training — so the knowledge stored in those parts stays intact.
Sam: So it's less about building a video model from scratch and more about carefully extending an image model without breaking what it already knows.
Alex: That's a good way to put it. The paper suggests that combining both strategies — the guidance signal and the freezing — preserves image ability much better than video training alone.
Sam: And what about actually running this efficiently? A sparse model with lots of routing logic could easily end up slower than just using a bigger dense model.
Alex: That's where the implementation work comes in. The standard way to handle sparse routing on a GPU involves a lot of back-and-forth data movement, which creates a bottleneck. The authors rewrote that step using what are called fused operations — instead of doing things one at a time and shuffling data around between each step, they bundle the work together so the GPU handles it in fewer, more efficient passes.
Sam: And that made a meaningful difference in speed?
Alex: It did. With that change, the model ran at roughly three-quarters the latency of a stronger dense baseline, while maintaining similar accuracy. So you're getting comparable quality at noticeably lower cost.
Sam: So the paper isn't just proposing an architecture — it's also showing that the architecture can actually run in practice.
Alex: Right. That's an important part of the contribution. A design that's theoretically efficient but slow in practice doesn't help anyone. The paper is careful to show that the sparse approach works at the implementation level, not just on paper.
Sam: What's the takeaway, then? Is this a solved problem, or more of a step in a direction?
Alex: The authors are measured about it. The paper shows that sparse vision encoders can scale, stay practical, and retain image understanding when extended to video. It's a careful step, not a final answer. But it does point to a useful direction — that the right architecture choices, combined with thoughtful training strategies, can make vision models more capable without simply making them more expensive to run.
Sam: That's a more useful result than it might sound at first.
Alex: It is. The question of how to scale vision models efficiently matters a lot as these systems get used in more demanding settings. Thanks for listening to ResearchPod.