Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a paper about a smarter way to build the part of an AI model that looks at images and video — the component that turns raw pixels into something the model can actually think about.
Sam: And the basic problem is that making that component bigger usually makes it slower and more expensive to run?
Alex: Exactly. With a standard design, every image has to pass through every part of the model, every time. So as you add more capacity, the cost goes up in lockstep. This paper asks whether you can break that link — get more knowledge into the model without making every image pay the full price.
Sam: So instead of one big system that does everything, you'd have a collection of specialists, and each image only visits a few of them?
Alex: That's the core idea. Imagine a school where instead of one teacher who covers every subject, you have dozens of specialists. Each student only visits the teachers relevant to what they're studying that day. The model works similarly — it has a pool of specialist submodules, and each piece of an image gets routed to just a small subset. That way, the total knowledge in the system can be large, but the active work at any moment stays manageable.
Sam: That kind of routing system — does it have a name?
Alex: It's called a mixture of experts. It's been used in language models for a while, but vision is a different challenge. The authors found that you can't just copy the language model design and expect it to work well on images. Visual data is much more varied — textures, edges, motion, depth — and that variety changes what the optimal layout looks like.
Sam: What did they find worked better for vision specifically?
Alex: Smaller, more focused specialists. Think of it like the difference between a generalist doctor and a team of narrow specialists. A coarse setup wastes capacity because one large block has to handle too many different kinds of visual detail. A finer layout lets each part focus on something more specific, and the paper found that this beats both a simple dense model and a rougher sparse design.
Sam: But if the model is only activating a few specialists at a time, what stops it from just always using the same popular ones and ignoring the rest?
Alex: That's a real risk. If one specialist becomes the go-to for everything, the others never develop, and you've wasted the whole point of having many of them. The paper handles this with a balancing adjustment built into the routing step itself. The router keeps track of how busy each specialist already is. Busy ones get nudged down in priority, underused ones get nudged up. It's a bit like a teacher redistributing students so no one's classroom is overcrowded while others sit empty — and it works without adding a separate training penalty.
Sam: Okay, so that handles the image side. What about video? I'd have thought a model good at images would naturally extend to video.
Alex: You'd think so, but it's more complicated. Video asks the model to handle something new — motion, changes across frames, relationships between moments in time. When you train on video, that pressure can pull the model away from the image patterns it already learned. It's a bit like a musician who's great at classical piano, then spends months learning jazz. They can start to lose the precision they had before.
Sam: So how do you stop that drift?
Alex: Two ways. First, the paper keeps the original image-trained model running alongside the new one during video training, and compares their outputs. If the video model's frame-by-frame understanding starts drifting away from what the image model would have said, that acts as a corrective signal. Second, some of the specialist parts get frozen — their weights don't change during video training — so the knowledge stored in those parts stays intact.
Sam: So it's less about building a video model from scratch and more about carefully extending an image model without breaking what it already knows.
Alex: That's a good way to put it. The paper suggests that combining both strategies — the guidance signal and the freezing — preserves image ability much better than video training alone.
Sam: And what about actually running this efficiently? A sparse model with lots of routing logic could easily end up slower than just using a bigger dense model.
Alex: That's where the implementation work comes in. The standard way to handle sparse routing on a GPU involves a lot of back-and-forth data movement, which creates a bottleneck. The authors rewrote that step using what are called fused operations — instead of doing things one at a time and shuffling data around between each step, they bundle the work together so the GPU handles it in fewer, more efficient passes.
Sam: And that made a meaningful difference in speed?
Alex: It did. With that change, the model ran at roughly three-quarters the latency of a stronger dense baseline, while maintaining similar accuracy. So you're getting comparable quality at noticeably lower cost.
Sam: So the paper isn't just proposing an architecture — it's also showing that the architecture can actually run in practice.
Alex: Right. That's an important part of the contribution. A design that's theoretically efficient but slow in practice doesn't help anyone. The paper is careful to show that the sparse approach works at the implementation level, not just on paper.
Sam: What's the takeaway, then? Is this a solved problem, or more of a step in a direction?
Alex: The authors are measured about it. The paper shows that sparse vision encoders can scale, stay practical, and retain image understanding when extended to video. It's a careful step, not a final answer. But it does point to a useful direction — that the right architecture choices, combined with thoughtful training strategies, can make vision models more capable without simply making them more expensive to run.
Sam: That's a more useful result than it might sound at first.
Alex: It is. The question of how to scale vision models efficiently matters a lot as these systems get used in more demanding settings. Thanks for listening to ResearchPod.