Ellis Brown, Chris Hoang, Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu
8 min
How to best encode visual inputs in multimodal large language models has remained an open question. Existing approaches have typically bundled three design choices together: using a pretrained vision encoder, assigning dedicated parameter capacity to vision tasks, and placing that capacity before the multimodal fusion point. Because previous models differed in architecture, data, and training compute, it has been impossible to isolate the true impact of these individual choices. This paper presents a controlled, from-scratch study that unbundles these components by training each model on identical data and at matched lifetime compute.
The authors identify three primary results that challenge current conventions in multimodal design:
By disentangling these design choices, this research provides a simplified, more efficient blueprint for building multimodal systems. The evidence suggests that researchers can move away from the overhead of pre-training separate vision encoders and toward architectures that integrate dedicated vision capacity directly into the multimodal backbone. This not only simplifies the training pipeline but can also improve performance on visual tasks like OCR and chart reading while lowering the time-to-first-token for inference.
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.
Sam: [steady and precise] They call it “lifetime compute”: computation across the training stages that produce the model. Encoder pretraining counts, rather than arriving as a free advantage. Their accounting excludes the later instruction-tuning stage used for evaluation.
Alex: [reflective and direct] So the question becomes how best to allocate one project budget, not just how good the final training stage looks.
Sam: [explaining with focus] Yes. They also test the practical alternative, where encoder pretraining is treated as free. But first, the shared budget lets them ask whether separate visual pretraining earns its cost. [[RP_SECTION:dedicated-vision-capacity|Dedicated Vision Capacity]]
Alex: [curious and measured] What happens when they remove only that separate stage, keeping the image-processing network?
Sam: [plain and confident] An encoder trained jointly from scratch performs on par with pretrained encoders. They test several pretraining objectives, including matching images with text and generating captions. Spending a large share of the budget upstream can hurt, because less remains for joint training.
Alex: [skeptical but calm] Perhaps the pretrained encoders simply weren’t good enough. Does stronger pretraining help when its cost doesn’t count?
Sam: [careful and clear] Longer pretraining improves image recognition, but not the average score on visual questions. With equal subsequent multimodal training, a scratch-trained encoder matches the pretrained alternative. Better performance on the encoder’s recognition test does not automatically transfer to answering questions about images.
Alex: [summarizing with care] That separates a useful visual representation for recognition from one that helps this whole system answer questions. What happens if they remove the encoder itself?
Sam: [focused and explanatory] Then image patches enter the shared backbone directly. That encoder-free design performs worse on visual understanding, while language scores remain comparable. Evaluation covers sixteen visual question-answering benchmarks, including general questions, knowledge, reading text and charts, and vision-focused tasks.
Alex: [probing and direct] But removing an encoder also removes parameters. Could this just mean the smaller model does worse?
Sam: [precise and unhurried] They test that objection. In one matched-budget comparison, the scratch encoder scores about forty-four percent on their visual-question average. The encoder-free model scores thirty-nine percent. Adding the missing parameter budget back into a wider or deeper shared backbone does not close the gap.
Alex: [thoughtful and steady] The extra shared parameters do help language scores, though. So they aren’t useless; they’re less effective for the visual task.
Sam: [measured and explanatory] That’s what the comparison shows. The remaining visual gap is largest for reading text and charts. They also try richer ways of converting patches into inputs, without closing the gap at the standard training resolution.
Alex: [careful and questioning] That last qualification matters. Is there a setting where one of those alternatives does catch up?
Sam: [plain and precise] A hierarchical patch-processing network matches the encoder when given much higher-resolution images compressed into the same input length. So the evidence supports dedicated vision capacity in the main setup, not a claim that every possible encoder-free design must fail.
Alex: [engaged and measured] Now the placement question. If dedicated visual processing helps, why should it have to finish before text and images meet? [[RP_SECTION:integrating-visual-processing|Integrating Visual Processing]]
Sam: [explaining with focus] It may not. The authors define fusion as the first layer where image and text representations can interact. They move the encoder’s parameter budget into vision-only branches inside the backbone. Every input still uses the shared machinery; image inputs also use those extra branches.
Alex: [inquiring and calm] Where do those branches work best? Spreading them throughout the network sounds like an obvious option.
Sam: [clear and steady] They compare spreading them, putting them late, and concentrating them early. The successful placement is the first eight layers. Spreading the same budget or placing it late does no better than the encoder-free model on average visual questions.
Alex: [thoughtful and probing] Does that establish why early processing matters, or just where it works?
Sam: [careful and measured] It establishes the placement result. The authors hypothesize that vision-specific processing helps most while image inputs are still close to raw patches. They also test letting text use those branches: reserving them for images works better.
Alex: [direct and curious] How far does this survive scaling? An architectural preference at one size could disappear in a larger model.
Sam: [steady and explanatory] Their scaling study spans backbones from roughly six hundred million to six billion parameters. Dedicated vision capacity grows alongside them. The inside-the-backbone design tracks the scratch encoder, while their fitted scaling relationships put encoder-free at about three times the computation to match visual performance.
Alex: [practical and measured] Most builders also start from an existing language model. Does learning everything together make this an unusual case?
Sam: [clear and reassuring] They test that extension using a language model they pretrained on text. Jointly learned vision capacity still works well, whether outside or inside the backbone. Encoder-free still trails, and the separately pretrained vision encoder trails the scratch encoder in that setting.
Alex: [reflective and direct] There’s also the speed claim from the opening. What should we take from that without assuming every deployment behaves alike? [[RP_SECTION:deployment-and-future-directions|Deployment and Future Directions]]
Sam: [careful and plain] In their serving benchmark, the internal vision branches reduce time to the first generated token compared with a separate encoder. The encoder must finish before the backbone starts. But mixed text-and-image request batching can change that trade-off, so the deployment matters.
Alex: [skeptical and measured] What is the biggest reason not to generalize the pretraining result too broadly?
Sam: [precise and sober] They pretrain encoders on the same images used for multimodal training. That does not rule out benefits from released encoders trained on additional data. They also test dense backbones, leaving sparse expert-based models, richer image data, and video as open directions.
Alex: [wrapping up calmly] Given those boundaries, who should read the full document, and where should they begin?
Sam: [decisive and warm] Researchers designing multimodal architectures or allocating training budgets should read it. Start with the dedicated-capacity comparison in Section Four, then the early-layer placement tests in Section Five. For adapting pretrained language models or serving image requests, go to Section Six.
Alex: [reflective and inviting] And for someone who only needs the central lesson?
Sam: [plain and memorable] In these experiments, removing the separate encoder works when you keep its visual job. Drop the boundary, not the dedicated capacity.
Alex: [calm and closing] Keep that distinction in mind when comparing multimodal models.