Xuehang Guo, Pengyuan Li, Tom Hope, Tirthankar Ghosal, Manling Li, Qingyun Wang
5 min
Cross-representation understanding—the ability to map between chart images, tabular data, and visualization code—is a critical bottleneck in modern AI. Current approaches often treat these as independent, supervised tasks, which is problematic because the relationships are inherently one-to-many (e.g., one chart can be represented by multiple valid tables or code snippets). This leads to high annotation costs and poor generalization across different task directions.
CoCoEvolve addresses these challenges by defining a principled cycle of representations: Chart (V) ↔ Table (T) ↔ Code (C). Instead of relying on scarce labeled data, the authors treat these three modalities as multiple, equivalent views of the same underlying data.
Key components of the framework include:
To overcome the limitations of existing "LLM-as-a-judge" methods, which often collapse complex visual and structural fidelity into a single, unreliable score, the authors introduce CoCoEvolve@Eval. This suite decomposes evaluation into interpretable, multidimensional metrics (e.g., structural, semantic, perceptual, and stylistic accuracy) using both rule-based rubrics and fine-grained LLM/MLLM-based assessment.
CoCoEvolve demonstrates that models can learn to reason across complex, multimodal representations without relying on expensive, manually curated datasets. By leveraging the natural semantic redundancy between charts, tables, and code, the framework achieves significant performance gains—up to 37.91% on non-overlapping test sets and 46.88% in out-of-domain settings—offering a more scalable path toward robust, cross-modal AI systems.
As chart images, tabular data, and visualization code play increasingly important roles across diverse domains, cross-representation understanding across these modalities poses fundamental challenges for AI systems: the relationships across representations are inherently \textit{one-to-many}, supervision is ambiguous and costly, and model optimization lacks a principled signal that is both direction-adaptive and representation-generalizable beyond task-specific objectives. We introduce CoCoEvolve to improve consistency across chart, table, and code representations. Instead of treating cross-representation mapping as a one-to-many problem, we define explicit one-to-one correspondences and optimize models using agreement between representations, without additional annotations. During training, CoCoEvolve@Train performs co-evolution across the chart-table-code cycle, while CoCoEvolve@Test applies the same consistency objective at inference time for test-time co-optimization. We also present CoCoEvolve@Eval, an evaluation suite covering all six cross-representation tasks. Across four benchmarks, CoCoEvolve improves performance in both training-time and test-time settings. Our project page: https://xhguo7.github.io/CoCoEvolve/.
Alex: That's a meaningful result. It sounds like they've taken the very thing that made this problem hard—all those different ways to represent the same data—and turned it into the engine of learning.
Sam: That's a good way to put it. The ambiguity that used to require human intervention is now what drives the self-correction. It's a meaningful shift in how you approach this kind of problem.
Alex: Which brings up a natural question—if the model is teaching itself, how do you actually grade it? How do you know when it's doing well?
Sam: That's the other half of the paper. Existing evaluation methods tend to give a single overall score, which turns out to be unreliable for something this nuanced. So the authors developed their own evaluation system alongside the training framework. Think of it like grading a student not just on their final answer, but on their entire process—did they use the right data? The correct chart type? The proper visual style?
Alex: So instead of one big grade, they break the task into smaller, independent pieces.
Sam: Exactly. They split the evaluation into five specific dimensions—things like data accuracy, axis labels, and overall layout. Each dimension gets its own score on a scale from zero to five. That gives researchers a clear picture of where a model is strong and where it's struggling, rather than a single number that hides all of that detail.
Alex: That also sounds much harder to "game." A model can't just get lucky on one thing and mask its weaknesses.
Sam: Precisely. And it revealed some useful patterns. Smaller models tended to stumble on basic data recognition—just reading the numbers off a chart correctly. Larger models handled that better, but sometimes failed on style, like getting colors or formatting wrong. Without those five separate categories, you'd never know which part of the chain was breaking down.
Alex: And is this evaluation system something that runs once at the end, or does it feed back into the training?
Sam: Both. They apply it during testing as well—which they call "CoCoEvolve@Test"—and doing so leads to further improvements. The evaluation isn't just a scorecard; it becomes another feedback signal that helps the model keep refining its understanding. It turns the grading rubric into a diagnostic tool.
Alex: So the whole system—training and evaluation—is built around the same core idea: use disagreement and mismatch as a signal, rather than waiting for a human to step in.
Sam: That's the through-line. By treating charts, tables, and code as a unified cycle rather than isolated tasks, and by building an evaluation system that's specific enough to actually diagnose failures, they've created something considerably more robust than what came before. It's a meaningful step toward AI that can reason across different data formats without needing constant human supervision.
Alex: Thanks for walking us through it, Sam. And thanks to everyone listening to ResearchPod.