Vision-language MoE batches contain different numbers of image and text tokens. Image resolution, image count, tiling, and prompt length all change this token mix. We call the standard token-level Switch auxiliary loss Std-Aux. Std-Aux balances only the mixed load, so large image and text load errors can cancel at one mix. On our main model, the same trained router shows more than a fivefold change in load imbalance across image resolutions. We hold the image and text load profiles fixed and derive the exact load curve as the token mix varies. The image-text load gap controls sensitivity to the token mix. Physical preprocessing can also change the conditional profiles. The fixed-profile law excludes such changes. To design a remedy, we examine the router input structure. Image and text occupy distinct regions, while visual tokens group strongly by source image. The modality boundary motivates separate image and text terms. The image boundary motivates one equal-weight routing instance per image. ReBA, or Relax Within, Balance Across, implements both choices. Across four split backbones, ReBA lowers load on every reported benchmark input while keeping mean task accuracy comparable to Std-Aux. ReBA also lowers average load over the tested range and worst physical load under resolution and tiling shifts. Code is available at https://github.com/ZiangWu-77/ReBA.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at why multimodal models—those handling both text and images—often slow down when you change input resolution.
Sam: We're discussing "ReBA"—short for "Relax Within, Balance Across." The puzzle is that these models become imbalanced during use, creating bottlenecks that weren't there during training.
Alex: So the paper asks why the way they distribute work across internal experts fails when the mix of images and text changes?
Sam: Exactly. Think of a large AI model as a team of specialists—what researchers call a "Mixture-of-Experts." The idea is that different specialists handle different parts of a task, and you want the workload spread evenly so no single specialist gets overwhelmed. The problem is that current systems only track the total workload, which hides the fact that images and text are handled very differently under the hood.
Alex: Wait, so the total looks fine on paper, but the individual parts are actually struggling?
Sam: Precisely. Imagine a restaurant where the manager only counts total orders for the whole kitchen. If the appetizer station is completely slammed while the main course station sits idle, the manager sees a balanced average and thinks everything is fine—until the menu shifts and the whole kitchen crashes. Standard training allows these imbalances to cancel each other out, right up until the workload changes in a way they can't compensate for.
Alex: It's an average hiding a real problem. So how does ReBA fix that?
Sam: The researchers found that image data and text data naturally occupy distinct regions in the model's internal logic—they cluster separately, like two different neighborhoods in a city. ReBA forces the model to balance the image workload and the text workload independently. By treating them separately, the system can no longer use a calm text station to mask a struggling image station.
Alex: And images are treated as "routing instances"—what does that mean in practice?
Sam: Think of a classroom where you want to make sure every student gets heard. If you just count total words spoken, a few talkative students dominate the discussion. But if you give every student exactly one vote, regardless of how much they've said, the balance is fairer. ReBA does something similar: it treats every image as one equal-weight unit, regardless of how many data points—or tokens—that image actually contains. A high-resolution image might generate far more tokens than a low-resolution one, but under ReBA, both count as a single vote in the routing decision.
Alex: So when you increase the resolution of an image, it doesn't suddenly gain outsized influence over the system?
Sam: Exactly. That's the core of why the model stays stable when resolution changes. By counting votes rather than raw tokens, the routing decisions don't get skewed just because one image happens to be more detailed than another.
Alex: It sounds like a meaningful improvement—not about more processing power, but better organization of the power you already have.
Sam: That's a fair way to put it. By aligning what the model is trained to optimize with the actual structure of the data it handles, they've created a more stable and predictable system. The experiments showed this lowers what the paper calls the "load gap"—the difference between how busy the busiest specialist is versus the least busy—consistently across different model sizes.
Alex: And does this better organization come at a cost to actual performance?
Sam: The paper reports that task accuracy remains comparable to the standard approach. The primary gain is in stability and efficiency—making sure resources are used as intended. It doesn't make the model more capable in the traditional sense, but it does make it more reliable under the kind of varied, real-world conditions where these systems are actually deployed.
Alex: So it's less about building a smarter system and more about making sure the system you've built behaves consistently—even when the inputs change in ways it didn't see during training.
Sam: That's precisely it. And for anyone building or deploying these systems at scale, that kind of predictability matters quite a lot. A model that performs well on average but collapses under certain input conditions is a much harder problem to manage than one that's simply somewhat less capable overall.
Alex: Thanks for walking us through that, Sam. And thank you for listening to ResearchPod.