Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs, yet existing benchmarks predominantly focus on pure-text queries. In real-world scenarios, language also frequently appears as visualized text embedded in images, raising the question of whether current VLMs handle such input requests comparably. We introduce VISTA-Bench, a systematic benchmark from multimodal perception, reasoning, to unimodal understanding domains. It evaluates visualized text understanding by contrasting pure-text and visualized-text questions under controlled rendering conditions. Extensive evaluation of over 20 representative VLMs reveals a pronounced modality gap: models that perform well on pure-text queries often degrade substantially when equivalent semantic content is presented as visualized text. This gap is further amplified by increased perceptual difficulty, highlighting sensitivity to rendering variations despite unchanged semantics. Overall, VISTA-Bench provides a principled evaluation framework to diagnose this limitation and to guide progress toward more unified language representations across tokenized text and pixels. The source dataset is available at https://github.com/QingAnLiu/VISTA-Bench.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at how AI models that handle both pictures and words sometimes stumble in unexpected ways.
Sam: These are called vision-language models, or VLMs for short. Imagine a smart assistant that reads books and looks at photos at the same time—it's built by adding picture-processing parts to big text-based chat systems, so it can answer questions about what's in an image or combine words with visuals.
Alex: That sounds useful for real life, like analyzing a screenshot. But do they handle text inside images just as well as plain typed words?
Sam: Not always. In everyday situations, words often show up printed on signs, charts, or screenshots—turning them into pictures that the model has to visually scan and read, rather than process directly as typed letters. The study found a clear drop in performance: on one key test of knowledge questions, a top model went from about 76% accurate with plain text to 68% when the same questions were shown as images. Researchers call this difference a 'modality gap'—it shows the model doesn't treat picture-text the same as direct text, even though the meaning is identical.
Alex: So even high-performing models can lose accuracy just because the words are pixels instead of tokens? What's a real-world example of that going wrong?
Sam: Picture an AI helper looking at a medical screenshot with dosage instructions. Tiny changes in how the font looks—like slight blurring or style shifts—can make it misread the drug name or amount, picking the wrong option. This gap grows with tougher visuals, like small or fancy fonts.
Alex: That makes sense why it matters for reliable AI. So they built something called VISTA-Bench to test this properly?
Sam: Yes—it's a benchmark with 1,500 matched questions, half plain text and half rendered as controlled images, covering perception, reasoning, and knowledge tasks. The key is a careful rendering process using LaTeX to handle text, code, and formulas accurately, then checking fidelity with another AI judge. This isolates whether the problem is just poor 'reading' from pixels.
Alex: So this benchmark breaks things down into specific types of tasks—like testing how the model sees, thinks, and recalls info when text is pixels. Walk me through what those main categories test, in simple terms.
Sam: One category looks at basic seeing and spotting details in a scene where text is part of the picture, like finding a sign in a photo or noting its color—things that mix vision with reading pixels. Another tests step-by-step thinking, such as figuring out relationships between objects described in pixel-text, like 'if this box is left of that one, what's next?' A third checks if the model pulls stored facts from pixel-text plus images, say medical info from a chart. The toughest isolates just reading facts from pixels alone, no extra pictures.
Alex: That last one sounds like a real stress test for pure text-reading from images. How did the models actually do across these—any patterns stand out?
Sam: The drop-off showed up everywhere, but hit hardest on reasoning and knowledge tasks—up to 30% for some models. Perception held up better, suggesting models can spot basics but struggle chaining ideas when text is visual. Overall, stronger models narrowed the gap to around 2%, but no one fully closed it. The paper points to weak pixel-reading as the main bottleneck.
Alex: Reasoning amplifies small reading slips into big errors. Why does font size or style make such a difference there?
Sam: Imagine trying to read tiny print on a blurry photo versus big clear letters—smaller or fancy fonts create visual glitches that trip up the model's scanning, turning correct knowledge into misreads. Tests on one model showed gaps jumping to 6-8% with 9-point text or brush styles, versus near-zero on large clean fonts. Clean pixels sometimes even edged out pure text slightly, hinting images can ground answers better if reading works. Extra scene pictures in multi-image tasks cut the gap too, by giving context to fix pixel errors.
Alex: So surrounding visuals act like clues to patch reading mistakes. But this is just open models—what about limits?
Sam: Right—the study sticks to open-source ones, skipping closed giants like upcoming GPT versions due to access issues. It focuses on multiple-choice, not open answers, and suggests agent tools or generation tests next.
Alex: So the push is toward models that see pixel-text and typed text the same way. How exactly did they create those visualized versions without introducing errors?
Sam: They started by cleaning the text carefully, like proofreading a handwritten note to fix curly quotes into straight ones and shield code snippets so special symbols don't break during printing. Next, they spotted math parts or symbols—like Greek letters or degree signs—and rewrote them into precise formats that print perfectly. This prep work feeds into a tool called LaTeX, which is like a super-accurate recipe for laying out text and formulas on a page, ensuring everything looks sharp and readable. They set a fixed width of 800 pixels for every image, like choosing a standard paper size so lines wrap the same way. Fonts varied across common styles—from clean ones to scripty handwriting—and sizes from tiny 9-point to huge 48-point, mimicking real screenshots or scans.
Alex: That handles variety without cheating the comparison. But how did they confirm these images truly matched the original text?
Sam: They used another AI model as a judge to inspect each one side-by-side with the source text. It checked if every character, word spacing, code format, and formula rendered perfectly on a strict 0-to-2 scale—2 for flawless, down to 0 for clear mistakes. Only perfect scores made the cut; others got tweaks to fonts or layout and rechecked until ideal.
Alex: Iterative fixing by AI judge keeps fidelity high. Did the way they phrased questions to the models affect how well they handled these pixels?
Sam: Yes—they tried prompts from super-short, like 'read and pick A/B/C/D,' to longer ones urging careful reading of options, or even step-by-step thinking without explaining aloud. Detailed 'think step-by-step' versions narrowed the gap a bit for some models, suggesting clearer guidance helps decoding pixels. Still, no prompt fully erased the difference, pointing to deeper visual processing limits.
Alex: So pulling this together, the tests show models consistently drop accuracy by 2 to 30% when reading the same questions as images instead of plain text, with bigger hits on tougher reasoning and knowledge tasks. And factors like font style or prompt wording make it worse or better in predictable ways. What are the main patterns from the top open models?
Sam: The paper outlines three main patterns from evaluating models like Qwen3-VL-8B-Instruct and InternVL-3.5-8B. First, the gap persists across tasks, showing no model fully equates pixels with direct text yet. Second, steadiness at reading varies with visual details like small fonts or script styles, and on how instructions are worded—clearer prompts help a little. Third, adding extra pictures in combined tasks cuts the gap somewhat, as visuals give clues to fix small reading slips. It covers open-source models with standard tools from kits like VLMEval, drawing from benchmarks on knowledge and multimodal tasks. Closed models aren't tested due to access, and it sticks to picking answers from options.
Alex: Fair point—it grounds the work without overclaiming. Practically, this pushes for training that makes pixel-text and typed text equivalent, so AI handles scans or screenshots reliably.
Sam: Yes, the upshot is smoother processing for all language as visual input, aiding things like document scanning. It's a notable step exposing where vision-language setups fall short, motivating unified designs that don't distinguish how text arrives. The benchmark offers a solid tool to track progress objectively.
Alex: That wraps up a clear picture of the challenges and paths forward in making these models more consistent across text formats. Thanks to Sam for breaking it down so thoughtfully. That's it for this look at how AI reads visualized text—thanks for listening to ResearchPod.