M. Brossier, T. Isenberg, K. Schönborn, J. Unger, M. Romero, J. Björklund, A. Ynnerman, L. Besançon
9 min
Abstract
We report on a systematic, PRISMA-guided survey of research at the intersection of LLMs and visualization, with a particular focus on visio-verbal interaction -- where verbal and visual modalities converge to support data sense-making. The emergence of Large Language Models (LLMs) has introduced new paradigms for interacting with data visualizations through natural language, leading to intuitive, multimodal, and accessible interfaces. We analyze 48 papers across six dimensions: application domain, visualization task, visualization representation, interaction modality, LLM integration, and system evaluation. Our classification framework maps LLM roles across the visualization pipeline, from data querying and transformation to visualization generation, explanation, and navigation. We highlight emerging design patterns, identify gaps in accessibility and visualization reading, and discuss the limitations of current LLMs in spatial reasoning and contextual grounding. We further reflect on evaluations of combined LLM-visualization systems, highlighting how current research projects tackle this challenge and discuss current gaps in conducting meaningful evaluations of such systems. With our survey we aim to guide future research and system design in LLM-enhanced visualization, supporting broad audiences and intelligent, conversational interfaces.
Alex: Okay, so not just typing—speech and gestures too. But does the type of chart matter for how well the AI handles it?
Sam: Yes, they categorize visual representations into familiar charts like bar graphs from tables, spatial ones such as maps, networks of connected nodes, images or videos, and custom oddities. Charts suit language models well since they're labeled clearly, like reading a scoreboard. Spatial maps challenge them most—the AIs struggle to reason about positions and layouts, much like trying to describe a room from a blurry photo.
Alex: Huh, that spatial gap stands out. And where are these systems mostly applied?
Sam: Mostly data science, then medicine and biology, physics or engineering, and education or social sciences. Hands-free needs drive speech in surgery, for example, while education benefits from simple explanations for beginners. The paper notes data science dominates as it's broad, but interdisciplinary mixes are rare due to small sample sizes.
Alex: Right, so evaluation—how do they test if these interactions actually help?
Sam: They track user studies with participant counts and expertise levels, plus metrics like task time, errors, or preferences via surveys. System checks cover accuracy, speed, and benchmarks. This highlights gaps, like needing better ways to measure combined language-visual performance.
Alex: So the classification uncovers where to improve—like better spatial tools or memory for long sessions.
Sam: Precisely. Techniques like retrieval-augmented generation—where the AI pulls exact data from storage to avoid inventing facts—plus chain-of-thought prompting for step-by-step reasoning, help address hallucinations and build reliable memory over time.
Alex: So those techniques—how do they play out across the actual steps of handling data visuals, like from grabbing info to making sense of it?
Sam: The report maps language models onto a sequence of jobs in the visualization process: pulling raw data, reshaping it, picking how to show it on screen, moving around the view, and explaining what it means. Think of it like cooking: first fetch ingredients from the pantry—that's data retrieval, often triggered by typing a question that turns into a search command like a database query. Most studies use text inputs for this, but few build long-term memory to remember past pulls.
Alex: Okay, fetching is basic—but what about reshaping the data next?
Sam: Reshaping, or data transformation, means cleaning and summarizing raw info, like sorting fruits by color or averaging sizes before plotting. Systems handle this by having the AI write short code snippets, create structured lists of steps, or plan multi-part fixes—like chopping veggies one by one then mixing. Text prompts drive most of this, with widgets for tweaks adding precision.
Alex: And turning that into actual charts?
Sam: Yes, visual encoding picks the display style—bars for comparisons, lines for trends—like choosing a pie chart for slices of a budget. AIs generate code or specs well for simple 2D charts, aided by examples in their training, but falter on custom colors or 3D scenes due to less practice data. Combining text with sliders lets users refine quickly.
Alex: Huh, so navigation follows—moving the view around?
Sam: Navigation adjusts the angle, like zooming on a map spot or rotating a 3D model. Speech shines here for hands-free shifts, as in surgery or VR, pairing with points for "zoom here." This task breakdown across 48 papers reveals patterns like text dominance and gaps in memory.
Alex: Okay, navigation makes sense for exploring. But the last piece—turning all that into real understanding?
Sam: That's sense-making, where you pull insights from the visuals, such as spotting trends or surprises in a chart. Systems use language models to give tailored explanations, like answering "why did sales spike here?" by linking data points to reasons. But the paper cautions careful use, as these AIs can hallucinate by inventing facts or agree too much with user biases.
Alex: Huh, so explanations via chat or speech, but with those risks flagged. How do they mix in pointing or drawing for better insights?
Sam: Text queries dominate, often with retrieval to ground answers in real data, but speech setups add spoken replies plus camera moves for context. Gestures shine in tools where sketching a chart area triggers an explanation you can edit with widgets.
Alex: Right, so across the whole pipeline, text leads but multimodals add precision where words fall short.
Sam: Exactly. Most papers stick to simple 2D charts since they're straightforward for AIs, unlike tricky spatial maps. Gaps persist in long-term memory and evaluating mixed inputs.
Alex: That frames the strengths and limits clearly—like chat for big ideas, gestures for spots. How do they make these LLMs more reliable overall?
Sam: One way is splitting work among multiple specialized AIs, like a team where one handles planning, another checks safety, and a boss coordinates—like dividing a big school project so experts tackle parts. This "mixture of agents" approach cuts errors by feeding each only needed info.
Alex: Huh, a team of AIs instead of one doing everything. Does that fix issues like forgetting past chats?
Sam: Yes, they add long-term memory, storing facts, how-to steps, or past talks for reuse—like a notebook you flip back to instead of starting blank each time.
Alex: Okay, so memory and teams build steadiness. But testing these setups—how do they check if it really works?
Sam: About half the studies test with users, averaging around 14 participants—mix of experts and novices—measuring task speed, accuracy, or preferences. The paper notes challenges: LLMs give varying outputs even on same inputs, so pure number scores fall short—needing user trust checks for hallucinations too. No tailored benchmarks for these chat-visual mixes yet.
Alex: So pulling it all together, this classification across the pipeline stages maps out where language models shine and stumble—like a blueprint for building better chat-based data tools.
Sam: Exactly. By breaking down tasks into analyst, author, and reader roles—from data pulls to sense-making—it reveals consistent patterns across those 48 papers, like text's dominance and multimodal synergies for precision. The paper positions this as a foundation for designing more reliable systems.
Alex: Right, and those evaluation challenges—no standard benchmarks yet. What about other limits in how they pulled this survey together?
Sam: A notable one is including preprints from arXiv, which keeps the review current but risks unpeer-reviewed work. Many papers omit key details on which language model version they used, complicating replication.
Alex: Huh, so transparency gaps could make comparisons shaky. Does the paper suggest fixes?
Sam: Yes—it calls for field-wide reporting standards on model details, prompts, and versions to boost reproducibility.
Alex: That's a clear picture of where the field stands. Thanks, Sam—this has been a thoughtful dive into LLM-enabled visualization interactions.
Sam: My pleasure, Alex.
Alex: Thanks for listening to ResearchPod.