Zhaoyan Sun, Shan Zhong, Daizhou Wen, Jiaxing Han, Guoliang Li, Ying Yan, Peng Zhang, Yu Su, Xiang Qi, Baolin Sun, Chengyuan Yang, Tao Fang, Huaiyu Ruan
5 min
Data science agents powered by Large Language Models (LLMs) aim to automate end-to-end workflows, from raw data processing to insight extraction. However, existing benchmarks often lack the complexity of real-world business scenarios, provide only coarse-grained performance metrics, and fail to capture the step-by-step reasoning required for data science. AgenticDataBench addresses these gaps by providing a rigorous, skill-centric evaluation framework.
The authors introduce a hierarchical skill framework, identifying 433 recurring data-centric operational patterns (e.g., Handling Missing Data, Feature Engineering) extracted from Stack Overflow solutions and real-world business practices. The benchmark construction involves three key steps: (1) extracting a representative skill hierarchy using embedding-based clustering and LLM refinement; (2) selecting skill-diverse tasks from real-world fintech datasets to maximize coverage; and (3) generating realistic tasks for public datasets using an LLM-based pipeline that ensures controlled skill composition. This allows for fine-grained analysis of agent performance at the skill level rather than just aggregate scores.
The empirical study of state-of-the-art agents (e.g., Smolagents, DA-Agent, Claude Code, CodeX) yields several insights. First, general-purpose agent harnesses generally outperform specialized data science agents due to more mature engineering optimizations, though they often incur higher token costs. Second, there is no single "best" agent; performance depends heavily on the synergy between the LLM and the harness. Third, "Data Analysis" is the most common category for agent failure, often due to poor handling of heterogeneous data or errors in statistical reasoning. Finally, the study highlights that increasing execution budgets (like time limits) rarely fixes performance issues, suggesting that failures are rooted in reasoning and planning rather than simple resource constraints.
AgenticDataBench provides a standardized, granular way to diagnose the specific weaknesses of data agents. By moving beyond simple pass/fail metrics, researchers can identify whether an agent struggles with data cleaning, modeling, or visualization, enabling more targeted improvements in agent architecture and training.
Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society. Automating this process is essential to reducing labor-intensive efforts for data scientists and enabling scalable data-driven applications. Recently, large language model (LLM)-based data agents have emerged as a promising solution to automate data science workflows. However, the field lacks comprehensive benchmarks to rigorously evaluate these agents across diverse scenarios with fine-grained granularity. To address this gap, we propose AgenticDataBench, a comprehensive benchmark featuring realistic tasks spanning diverse domains with fine-grained ground-truth labels. This enables evaluations to capture the diversity and complexity of data science workflows and the detailed performance of agents. First, to cover diverse domains, we collect real datasets and tasks from 15 vertical domains, including 5 real-world B2B use cases from a leading fintech company. Second, to remove redundancy in real-world tasks and generate high-quality tasks for domains lacking real data, we introduce data science skills, recurring data-centric operational patterns, and quantify benchmark coverage by the number of skills included. Representative skills are extracted from large-scale task solutions on Stack Overflow using skill-aligned hierarchical clustering. Third, for real-world business tasks, we select task-solution pairs that maximize diversity in skill composition, ensuring broad coverage of practical scenarios. Fourth, to generate realistic tasks for devise domains without real tasks, we propose a systematic LLM-based task generation approach to create workflows and tasks based on these skills. Finally, we evaluate state-of-the-art data agents using our annotated benchmark and open-sourced testbed, providing detailed skill-level insights.
Sam: So they're not just throwing random problems at the agent. They're deliberately choosing tasks to probe every corner of that skill map.
Alex: Right. And the grading works the same way — deliberately, not subjectively. Rather than asking one AI model to judge another, the system relies on execution feedback. If the agent's code actually runs and produces the correct result, it scores well. If the code crashes, times out, or returns wrong answers, the score drops. The outcome of the code running against real data is what matters — not whether the code looks good to another model.
Sam: So it's grounded in something concrete — the code either works or it doesn't.
Alex: Precisely. They also filter out skills that appear too rarely in the data, so every skill being tested has enough examples to produce a statistically meaningful result rather than a one-off fluke.
Sam: And what did all this actually reveal? Where do these agents tend to struggle?
Alex: This is where the findings get interesting. You might expect agents to stumble most on the complex modeling work — building predictive models, that sort of thing. But the largest share of failures actually came from what the paper calls "Data Analysis" tasks. Things like calculating basic statistics or summarising a dataset.
Sam: That's unexpected. Why is the analysis phase causing so many problems?
Alex: The paper suggests it comes down to a mismatch between the agent's logic and the environment it's operating in. Agents frequently hit time limits when working with large files, or they encounter a small syntax error — a missing comma, a misplaced bracket — and instead of correcting it and moving on, they get stuck in unproductive loops.
Sam: So it's not that the AI lacks the intelligence to do the analysis. It's that it gets derailed by friction — file sizes, small errors, system constraints.
Alex: That's the key insight. And notably, simply giving the agents more time to work didn't solve the problem. It just led to longer loops that still went nowhere. The bottleneck isn't raw capability — it's the interaction between the agent's decision-making and the system it's running inside.
Sam: So the implication is that we need to stop thinking of these agents as self-contained brains and start thinking of them as systems — where the environment matters just as much as the model.
Alex: That's the core takeaway. By identifying these specific failure modes, researchers can now build agents designed to handle these operational pitfalls — rather than just hoping a more powerful model will somehow figure it out on its own. The shift from a simple pass-or-fail score to a detailed skill map is what makes that kind of targeted improvement possible.
Sam: It's a much more rigorous way to hold these systems accountable — and to actually make them better.
Alex: It is. Diagnostic evaluation gives developers a map of where to look, rather than leaving them to guess. Thanks for listening to ResearchPod.