Scalable Visual Pretraining for Language Intelligence | Yiming Zhang et al. | ResearchPod

Transcript: Scalable Visual Pretraining for Language Intelligence

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a study that challenges how we train artificial intelligence. Sam, what's the core puzzle here?

Sam: So, most AI models today are trained on enormous amounts of text. The assumption baked into that approach is that all meaningful knowledge can be written down as a sequence of words. This paper pushes back on that. It argues that when we convert scientific documents into plain text, we're quietly discarding some of the most important information they contain.

Alex: So we're essentially starving our models — forcing them to read a description of a map instead of looking at the map itself?

Sam: That's a good way to put it. Think about a physics textbook. It doesn't just have words — it has diagrams, equations laid out in specific ways, and figures that sit right next to the text they're explaining. When you strip all of that down to plain text, you lose the spatial relationships. You lose the sense of where things are relative to each other on the page. The researchers argue that layout is itself a form of meaning. They call their alternative approach "Visual Pretraining." Instead of extracting the words from a document, they feed the raw page images directly into the model — letting it learn what they call the "visual grammar" of scientific writing.

Alex: But how does a model actually "read" an image? If it's not converting the image to text first, what is it doing?

Sam: Think about how you might learn to navigate a new city. You don't just memorize a list of street names — you build a mental picture of how the streets connect. The model does something similar with pages. It breaks each document image into a grid of small squares, called "patches" — think of it like cutting a photograph into a hundred small tiles. It then learns to predict what the next tile looks like, given all the tiles that came before it. By doing this repeatedly across thousands of documents, the model starts to internalize the patterns: where equations tend to appear relative to figures, how captions relate to graphs, how a proof is structured visually. The researchers call this "Autoregressive Visual Pretraining."

Alex: So it's not just passively looking at the image — it's actively trying to reconstruct it, which forces it to understand the structure underneath.

Sam: Exactly. And that understanding pays off. When the researchers compared this visual approach against traditional text-only training — using the exact same set of scientific documents — the models trained on raw page images consistently performed better on complex scientific reasoning tasks. What's particularly notable is the efficiency: the visual models achieved those better results using only about a quarter of the data that the text-only models required.

Scalable Visual Pretraining for Language Intelligence

Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, Haiteng Zhao, Tianyang Lin, Yunhua Zhou, Demin Song, Kuikun Liu, Haochen Ye, Haian Huang, Yuzhe Gu, Haijun Lv, Qipeng Guo, Bin Liu, Gaoang Wang, Kai Chen

Jul 10, 2026·5:34·arxiv·Original Paper

Visual PretrainingLanguage IntelligencePlatonic Representation HypothesisFoundation ModelsVisual Pretraining (VP)Text Pretraining (TP)

Listen

6 min

Briefing

Research Question

Modern foundation models are almost exclusively trained on text corpora, assuming that all essential knowledge can be represented as a linear sequence of tokens. This paper challenges this assumption, arguing that scientific documents contain critical information—such as complex equations, figures, and spatial layouts—that is inherently lost or distorted when documents are converted into plain text via OCR or parsing tools. The authors investigate whether training models directly on raw visual document pages can yield superior language and reasoning intelligence.

Approach

To test this, the authors introduce Visual Pretraining (VP), an unsupervised framework that trains models to predict document patches in a latent space without requiring explicit image-text pairing or text extraction. They conduct a systematic, controlled comparison between VP and traditional Text Pretraining (TP) using identical underlying scientific document corpora. By keeping the starting checkpoints and supervised fine-tuning (SFT) data constant, they isolate the impact of preserving the native visual structure of the documents.

Main Findings

Superior Reasoning: VP consistently outperforms text-only pretraining on scientific reasoning benchmarks (e.g., GPQA, AIME) across multiple model architectures, including Qwen and Llama.
Efficiency: VP is highly efficient, achieving better performance than TP while consuming only 25% of the token budget. This suggests that visual representations are a more compact and information-dense way to encode scientific knowledge.
Emergent Alignment: Without any labeled image-text pairs, VP improves cross-modal alignment, as evidenced by tighter clustering of visual and textual embeddings and improved performance on multimodal benchmarks like MMMU-Pro and ChartQAPro.
Structure Sensitivity: The performance gains are most significant on documents with high visual-structure density, confirming that the model is successfully internalizing knowledge from figures, tables, and complex layouts that text-based methods fail to capture.

Why It Matters

This work demonstrates that the "text-only" paradigm for foundation models is likely suboptimal for scientific and technical domains. By treating raw visual documents as a primary data source, researchers can unlock richer, more structured representations that enhance both logical reasoning and multimodal capabilities. This provides a scalable, efficient pathway for building more capable AI systems that can interpret the world as it is presented in technical literature.

Key Terms and Definitions

Visual Pretraining (VP) — An unsupervised learning paradigm where models are trained to predict visual document patches in latent space, preserving spatial and structural information.
Text Pretraining (TP) — The standard approach of converting visual documents into plain text (e.g., via parsing) before training, which often discards non-textual structural cues.
Visual-Structure Density — A metric representing the prevalence of figures, equations, tables, and complex layouts within a document page.
Linear CKA (Centered Kernel Alignment) — A similarity metric used to measure the agreement between the geometric structures of two different representation spaces.
Centroid Separation — A measure of cross-modal alignment where lower values indicate that visual and textual embeddings for the same content are closer together in the shared latent space.

Abstract

The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that language models must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.

Scalable Visual Pretraining for Language Intelligence: Research Paper Podcast