Unbundling Multimodal Vision Encoding

How to best encode visual inputs in multimodal large language models has remained an open question. Existing approaches have typically bundled three design choices together: using a pretrained vision encoder, assigning dedicated parameter capacity to vision tasks, and placing that capacity before the multimodal fusion point. Because previous models differed in architecture, data, and training compute, it has been impossible to isolate the true impact of these individual choices. This paper presents a controlled, from-scratch study that unbundles these components by training each model on identical data and at matched lifetime compute.

The authors identify three primary results that challenge current conventions in multimodal design:

• Separate Pretraining is Unnecessary: Models with encoders trained jointly from scratch with the language backbone performed on par with those using separate, complex pretraining, even when pretraining compute was treated as free.
• Dedicated Vision Capacity is Crucial: Models that removed the vision encoder entirely (encoder-free) consistently trailed behind on visual understanding tasks, even when their backbones were enlarged to match the encoder's total parameter count.
• Placement Flexibility: Dedicated vision capacity does not need to precede multimodal fusion. Integrating vision-only modules directly into the early layers of the shared backbone (in-fusion) achieved performance comparable to a pre-fusion encoder while potentially reducing inference latency.

By disentangling these design choices, this research provides a simplified, more efficient blueprint for building multimodal systems. The evidence suggests that researchers can move away from the overhead of pre-training separate vision encoders and toward architectures that integrate dedicated vision capacity directly into the multimodal backbone. This not only simplifies the training pipeline but can also improve performance on visual tasks like OCR and chart reading while lowering the time-to-first-token for inference.

Key Terms

Dedicated Vision Capacity

A set of parameters within a model reserved exclusively for processing vision-related tokens rather than being shared with text processing.

Lifetime Compute

The total training compute across all stages of a model's development, including pretraining of individual components like vision encoders, used to ensure fair comparison.

In-Fusion Capacity

A design approach where vision-only computation modules are inserted directly into the shared layers of the transformer backbone instead of residing in a separate, upstream encoder.

Encoder-Free Model

An architecture that maps raw image patches directly into a multimodal backbone without utilizing a distinct vision-specific transformation network.

Multimodal Fusion Boundary

The specific layer or point in a neural architecture where visual and textual information first interact and are jointly processed.

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.

Sam: [clear and explanatory] The contribution is separating the choices under common conditions. The authors train their own components, including the main language network, which they call the backbone. They share data sources and training settings, then compare designs at matched total training computation.

Alex: [thoughtful and questioning] Total training computation matters because a pretrained component has already consumed resources. How do they account for that? [[RP_SECTION:encoder-pretraining-and-compute|Encoder Pretraining and Compute]]

Sam: [steady and precise] They call it “lifetime compute”: computation across the training stages that produce the model. Encoder pretraining counts, rather than arriving as a free advantage. Their accounting excludes the later instruction-tuning stage used for evaluation.

Alex: [reflective and direct] So the question becomes how best to allocate one project budget, not just how good the final training stage looks.

Sam: [explaining with focus] Yes. They also test the practical alternative, where encoder pretraining is treated as free. But first, the shared budget lets them ask whether separate visual pretraining earns its cost. [[RP_SECTION:dedicated-vision-capacity|Dedicated Vision Capacity]]

Alex: [curious and measured] What happens when they remove only that separate stage, keeping the image-processing network?

Sam: [plain and confident] An encoder trained jointly from scratch performs on par with pretrained encoders. They test several pretraining objectives, including matching images with text and generating captions. Spending a large share of the budget upstream can hurt, because less remains for joint training.

Alex: [skeptical but calm] Perhaps the pretrained encoders simply weren’t good enough. Does stronger pretraining help when its cost doesn’t count?

Sam: [careful and clear] Longer pretraining improves image recognition, but not the average score on visual questions. With equal subsequent multimodal training, a scratch-trained encoder matches the pretrained alternative. Better performance on the encoder’s recognition test does not automatically transfer to answering questions about images.

Alex: [summarizing with care] That separates a useful visual representation for recognition from one that helps this whole system answer questions. What happens if they remove the encoder itself?

Sam: [focused and explanatory] Then image patches enter the shared backbone directly. That encoder-free design performs worse on visual understanding, while language scores remain comparable. Evaluation covers sixteen visual question-answering benchmarks, including general questions, knowledge, reading text and charts, and vision-focused tasks.

Alex: [probing and direct] But removing an encoder also removes parameters. Could this just mean the smaller model does worse?

Sam: [precise and unhurried] They test that objection. In one matched-budget comparison, the scratch encoder scores about forty-four percent on their visual-question average. The encoder-free model scores thirty-nine percent. Adding the missing parameter budget back into a wider or deeper shared backbone does not close the gap.

Alex: [thoughtful and steady] The extra shared parameters do help language scores, though. So they aren’t useless; they’re less effective for the visual task.

Sam: [measured and explanatory] That’s what the comparison shows. The remaining visual gap is largest for reading text and charts. They also try richer ways of converting patches into inputs, without closing the gap at the standard training resolution.

Alex: [careful and questioning] That last qualification matters. Is there a setting where one of those alternatives does catch up?

Sam: [plain and precise] A hierarchical patch-processing network matches the encoder when given much higher-resolution images compressed into the same input length. So the evidence supports dedicated vision capacity in the main setup, not a claim that every possible encoder-free design must fail.

Alex: [engaged and measured] Now the placement question. If dedicated visual processing helps, why should it have to finish before text and images meet? [[RP_SECTION:integrating-visual-processing|Integrating Visual Processing]]

Sam: [explaining with focus] It may not. The authors define fusion as the first layer where image and text representations can interact. They move the encoder’s parameter budget into vision-only branches inside the backbone. Every input still uses the shared machinery; image inputs also use those extra branches.

Alex: [inquiring and calm] Where do those branches work best? Spreading them throughout the network sounds like an obvious option.

Sam: [clear and steady] They compare spreading them, putting them late, and concentrating them early. The successful placement is the first eight layers. Spreading the same budget or placing it late does no better than the encoder-free model on average visual questions.

Alex: [thoughtful and probing] Does that establish why early processing matters, or just where it works?

Sam: [careful and measured] It establishes the placement result. The authors hypothesize that vision-specific processing helps most while image inputs are still close to raw patches. They also test letting text use those branches: reserving them for images works better.

Alex: [direct and curious] How far does this survive scaling? An architectural preference at one size could disappear in a larger model.

Sam: [steady and explanatory] Their scaling study spans backbones from roughly six hundred million to six billion parameters. Dedicated vision capacity grows alongside them. The inside-the-backbone design tracks the scratch encoder, while their fitted scaling relationships put encoder-free at about three times the computation to match visual performance.

Alex: [practical and measured] Most builders also start from an existing language model. Does learning everything together make this an unusual case?

Sam: [clear and reassuring] They test that extension using a language model they pretrained on text. Jointly learned vision capacity still works well, whether outside or inside the backbone. Encoder-free still trails, and the separately pretrained vision encoder trails the scratch encoder in that setting.

Alex: [reflective and direct] There’s also the speed claim from the opening. What should we take from that without assuming every deployment behaves alike? [[RP_SECTION:deployment-and-future-directions|Deployment and Future Directions]]

Sam: [careful and plain] In their serving benchmark, the internal vision branches reduce time to the first generated token compared with a separate encoder. The encoder must finish before the backbone starts. But mixed text-and-image request batching can change that trade-off, so the deployment matters.

Alex: [skeptical and measured] What is the biggest reason not to generalize the pretraining result too broadly?

Sam: [precise and sober] They pretrain encoders on the same images used for multimodal training. That does not rule out benefits from released encoders trained on additional data. They also test dense backbones, leaving sparse expert-based models, richer image data, and video as open directions.

Alex: [wrapping up calmly] Given those boundaries, who should read the full document, and where should they begin?

Sam: [decisive and warm] Researchers designing multimodal architectures or allocating training budgets should read it. Start with the dedicated-capacity comparison in Section Four, then the early-layer placement tests in Section Five. For adapting pretrained language models or serving image requests, go to Section Six.

Alex: [reflective and inviting] And for someone who only needs the central lesson?

Sam: [plain and memorable] In these experiments, removing the separate encoder works when you keep its visual job. Drop the boundary, not the dedicated capacity.

Alex: [calm and closing] Keep that distinction in mind when comparing multimodal models.

Listen to the full podcast →