We report on a systematic, PRISMA-guided survey of research at the intersection of LLMs and visualization, with a particular focus on visio-verbal interaction -- where verbal and visual modalities converge to support data sense-making. The emergence of Large Language Models (LLMs) has introduced new paradigms for interacting with data visualizations through natural language, leading to intuitive, multimodal, and accessible interfaces. We analyze 48 papers across six dimensions: application domain, visualization task, visualization representation, interaction modality, LLM integration, and system evaluation. Our classification framework maps LLM roles across the visualization pipeline, from data querying and transformation to visualization generation, explanation, and navigation. We highlight emerging design patterns, identify gaps in accessibility and visualization reading, and discuss the limitations of current LLMs in spatial reasoning and contextual grounding. We further reflect on evaluations of combined LLM-visualization systems, highlighting how current research projects tackle this challenge and discuss current gaps in conducting meaningful evaluations of such systems. With our survey we aim to guide future research and system design in LLM-enhanced visualization, supporting broad audiences and intelligent, conversational interfaces.
Alex: Welcome to another episode of ResearchPod. Sam, what are we looking at today?
Sam: This is a report called "State of the Art of LLM-Enabled Interaction with Visualization" by Brossier and colleagues. It surveys research where large language models—AI systems trained on huge amounts of text to understand and create human-like language—help people interact with charts and graphs through everyday conversation.
Alex: So the core puzzle is making complex data tools easier for regular people, like turning sliders and menus into simple chat?
Sam: Yes. Traditional visualization software often requires expert skills, such as fiddling with filters to dig into sales data—a novice might struggle to find top products from last quarter. This report uses a structured review method called PRISMA, like a clear recipe for sifting through studies transparently. It started with 353 papers and narrowed to 48 key ones. The review maps how language models fit into handling data visuals, from pulling data to explaining insights. It focuses on "visio-verbal interaction," where talking or typing combines with looking at charts, much like pointing at a map while asking directions.
Alex: And that pipeline breaks down the steps someone takes with data visuals?
Sam: Precisely. They frame it around user roles: the analyst who fetches and tweaks raw data, the author who turns it into charts, and the reader who navigates and interprets them. Language models step in at each stage, like helping query data in plain words or guiding exploration. This reveals patterns in how these systems work—and gaps, such as weak spatial understanding in the AIs. The paper suggests this classification guides better designs for intuitive interfaces.
Alex: Huh, so it's building a map of where these AIs fit best. That practical challenge with novices makes sense—why force everyone to learn software tricks when chat feels natural?
Sam: That's the key insight. By spotting design patterns and limitations like AI hallucinations or poor grasp of chart layouts, it points to opportunities for more reliable, accessible tools.
Alex: So those limitations like hallucinations and spatial issues—how do they classify the ways people interact with these systems?
Sam: They break interactions into input types. The main ones are typing natural language questions, speaking them aloud, pointing or gesturing—like waving a hand to highlight a chart area—and traditional clicks on buttons or sliders. This mix mimics how people talk and point during real conversations. Language models handle words while gestures add precise location info. Researchers group these as input modalities to spot what works best together.
Alex: Okay, so not just typing—speech and gestures too. But does the type of chart matter for how well the AI handles it?
Sam: Yes, they categorize visual representations into familiar charts like bar graphs from tables, spatial ones such as maps, networks of connected nodes, images or videos, and custom oddities. Charts suit language models well since they're labeled clearly, like reading a scoreboard. Spatial maps challenge them most—the AIs struggle to reason about positions and layouts, much like trying to describe a room from a blurry photo.
Alex: Huh, that spatial gap stands out. And where are these systems mostly applied?
Sam: Mostly data science, then medicine and biology, physics or engineering, and education or social sciences. Hands-free needs drive speech in surgery, for example, while education benefits from simple explanations for beginners. The paper notes data science dominates as it's broad, but interdisciplinary mixes are rare due to small sample sizes.
Alex: Right, so evaluation—how do they test if these interactions actually help?
Sam: They track user studies with participant counts and expertise levels, plus metrics like task time, errors, or preferences via surveys. System checks cover accuracy, speed, and benchmarks. This highlights gaps, like needing better ways to measure combined language-visual performance.
Alex: So the classification uncovers where to improve—like better spatial tools or memory for long sessions.
Sam: Precisely. Techniques like retrieval-augmented generation—where the AI pulls exact data from storage to avoid inventing facts—plus chain-of-thought prompting for step-by-step reasoning, help address hallucinations and build reliable memory over time.
Alex: So those techniques—how do they play out across the actual steps of handling data visuals, like from grabbing info to making sense of it?
Sam: The report maps language models onto a sequence of jobs in the visualization process: pulling raw data, reshaping it, picking how to show it on screen, moving around the view, and explaining what it means. Think of it like cooking: first fetch ingredients from the pantry—that's data retrieval, often triggered by typing a question that turns into a search command like a database query. Most studies use text inputs for this, but few build long-term memory to remember past pulls.
Alex: Okay, fetching is basic—but what about reshaping the data next?
Sam: Reshaping, or data transformation, means cleaning and summarizing raw info, like sorting fruits by color or averaging sizes before plotting. Systems handle this by having the AI write short code snippets, create structured lists of steps, or plan multi-part fixes—like chopping veggies one by one then mixing. Text prompts drive most of this, with widgets for tweaks adding precision.
Alex: And turning that into actual charts?
Sam: Yes, visual encoding picks the display style—bars for comparisons, lines for trends—like choosing a pie chart for slices of a budget. AIs generate code or specs well for simple 2D charts, aided by examples in their training, but falter on custom colors or 3D scenes due to less practice data. Combining text with sliders lets users refine quickly.
Alex: Huh, so navigation follows—moving the view around?
Sam: Navigation adjusts the angle, like zooming on a map spot or rotating a 3D model. Speech shines here for hands-free shifts, as in surgery or VR, pairing with points for "zoom here." This task breakdown across 48 papers reveals patterns like text dominance and gaps in memory.
Alex: Okay, navigation makes sense for exploring. But the last piece—turning all that into real understanding?
Sam: That's sense-making, where you pull insights from the visuals, such as spotting trends or surprises in a chart. Systems use language models to give tailored explanations, like answering "why did sales spike here?" by linking data points to reasons. But the paper cautions careful use, as these AIs can hallucinate by inventing facts or agree too much with user biases.
Alex: Huh, so explanations via chat or speech, but with those risks flagged. How do they mix in pointing or drawing for better insights?
Sam: Text queries dominate, often with retrieval to ground answers in real data, but speech setups add spoken replies plus camera moves for context. Gestures shine in tools where sketching a chart area triggers an explanation you can edit with widgets.
Alex: Right, so across the whole pipeline, text leads but multimodals add precision where words fall short.
Sam: Exactly. Most papers stick to simple 2D charts since they're straightforward for AIs, unlike tricky spatial maps. Gaps persist in long-term memory and evaluating mixed inputs.
Alex: That frames the strengths and limits clearly—like chat for big ideas, gestures for spots. How do they make these LLMs more reliable overall?
Sam: One way is splitting work among multiple specialized AIs, like a team where one handles planning, another checks safety, and a boss coordinates—like dividing a big school project so experts tackle parts. This "mixture of agents" approach cuts errors by feeding each only needed info.
Alex: Huh, a team of AIs instead of one doing everything. Does that fix issues like forgetting past chats?
Sam: Yes, they add long-term memory, storing facts, how-to steps, or past talks for reuse—like a notebook you flip back to instead of starting blank each time.
Alex: Okay, so memory and teams build steadiness. But testing these setups—how do they check if it really works?
Sam: About half the studies test with users, averaging around 14 participants—mix of experts and novices—measuring task speed, accuracy, or preferences. The paper notes challenges: LLMs give varying outputs even on same inputs, so pure number scores fall short—needing user trust checks for hallucinations too. No tailored benchmarks for these chat-visual mixes yet.
Alex: So pulling it all together, this classification across the pipeline stages maps out where language models shine and stumble—like a blueprint for building better chat-based data tools.
Sam: Exactly. By breaking down tasks into analyst, author, and reader roles—from data pulls to sense-making—it reveals consistent patterns across those 48 papers, like text's dominance and multimodal synergies for precision. The paper positions this as a foundation for designing more reliable systems.
Alex: Right, and those evaluation challenges—no standard benchmarks yet. What about other limits in how they pulled this survey together?
Sam: A notable one is including preprints from arXiv, which keeps the review current but risks unpeer-reviewed work. Many papers omit key details on which language model version they used, complicating replication.
Alex: Huh, so transparency gaps could make comparisons shaky. Does the paper suggest fixes?
Sam: Yes—it calls for field-wide reporting standards on model details, prompts, and versions to boost reproducibility.
Alex: That's a clear picture of where the field stands. Thanks, Sam—this has been a thoughtful dive into LLM-enabled visualization interactions.
Sam: My pleasure, Alex.
Alex: Thanks for listening to ResearchPod.