Zhaoyan Sun, Shan Zhong, Daizhou Wen, Jiaxing Han, Guoliang Li, Ying Yan, Peng Zhang, Yu Su, Xiang Qi, Baolin Sun, Chengyuan Yang, Tao Fang, Huaiyu Ruan
5 min
Abstract
Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society. Automating this process is essential to reducing labor-intensive efforts for data scientists and enabling scalable data-driven applications. Recently, large language model (LLM)-based data agents have emerged as a promising solution to automate data science workflows. However, the field lacks comprehensive benchmarks to rigorously evaluate these agents across diverse scenarios with fine-grained granularity. To address this gap, we propose AgenticDataBench, a comprehensive benchmark featuring realistic tasks spanning diverse domains with fine-grained ground-truth labels. This enables evaluations to capture the diversity and complexity of data science workflows and the detailed performance of agents. First, to cover diverse domains, we collect real datasets and tasks from 15 vertical domains, including 5 real-world B2B use cases from a leading fintech company. Second, to remove redundancy in real-world tasks and generate high-quality tasks for domains lacking real data, we introduce data science skills, recurring data-centric operational patterns, and quantify benchmark coverage by the number of skills included. Representative skills are extracted from large-scale task solutions on Stack Overflow using skill-aligned hierarchical clustering. Third, for real-world business tasks, we select task-solution pairs that maximize diversity in skill composition, ensuring broad coverage of practical scenarios. Fourth, to generate realistic tasks for devise domains without real tasks, we propose a systematic LLM-based task generation approach to create workflows and tasks based on these skills. Finally, we evaluate state-of-the-art data agents using our annotated benchmark and open-sourced testbed, providing detailed skill-level insights.
Sam: So they're not just throwing random problems at the agent. They're deliberately choosing tasks to probe every corner of that skill map.
Alex: Right. And the grading works the same way — deliberately, not subjectively. Rather than asking one AI model to judge another, the system relies on execution feedback. If the agent's code actually runs and produces the correct result, it scores well. If the code crashes, times out, or returns wrong answers, the score drops. The outcome of the code running against real data is what matters — not whether the code looks good to another model.
Sam: So it's grounded in something concrete — the code either works or it doesn't.
Alex: Precisely. They also filter out skills that appear too rarely in the data, so every skill being tested has enough examples to produce a statistically meaningful result rather than a one-off fluke.
Sam: And what did all this actually reveal? Where do these agents tend to struggle?
Alex: This is where the findings get interesting. You might expect agents to stumble most on the complex modeling work — building predictive models, that sort of thing. But the largest share of failures actually came from what the paper calls "Data Analysis" tasks. Things like calculating basic statistics or summarising a dataset.
Sam: That's unexpected. Why is the analysis phase causing so many problems?
Alex: The paper suggests it comes down to a mismatch between the agent's logic and the environment it's operating in. Agents frequently hit time limits when working with large files, or they encounter a small syntax error — a missing comma, a misplaced bracket — and instead of correcting it and moving on, they get stuck in unproductive loops.
Sam: So it's not that the AI lacks the intelligence to do the analysis. It's that it gets derailed by friction — file sizes, small errors, system constraints.
Alex: That's the key insight. And notably, simply giving the agents more time to work didn't solve the problem. It just led to longer loops that still went nowhere. The bottleneck isn't raw capability — it's the interaction between the agent's decision-making and the system it's running inside.
Sam: So the implication is that we need to stop thinking of these agents as self-contained brains and start thinking of them as systems — where the environment matters just as much as the model.
Alex: That's the core takeaway. By identifying these specific failure modes, researchers can now build agents designed to handle these operational pitfalls — rather than just hoping a more powerful model will somehow figure it out on its own. The shift from a simple pass-or-fail score to a detailed skill map is what makes that kind of targeted improvement possible.
Sam: It's a much more rigorous way to hold these systems accountable — and to actually make them better.
Alex: It is. Diagnostic evaluation gives developers a map of where to look, rather than leaving them to guess. Thanks for listening to ResearchPod.