Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society. Automating this process is essential to reducing labor-intensive efforts for data scientists and enabling scalable data-driven applications. Recently, large language model (LLM)-based data agents have emerged as a promising solution to automate data science workflows. However, the field lacks comprehensive benchmarks to rigorously evaluate these agents across diverse scenarios with fine-grained granularity. To address this gap, we propose AgenticDataBench, a comprehensive benchmark featuring realistic tasks spanning diverse domains with fine-grained ground-truth labels. This enables evaluations to capture the diversity and complexity of data science workflows and the detailed performance of agents. First, to cover diverse domains, we collect real datasets and tasks from 15 vertical domains, including 5 real-world B2B use cases from a leading fintech company. Second, to remove redundancy in real-world tasks and generate high-quality tasks for domains lacking real data, we introduce data science skills, recurring data-centric operational patterns, and quantify benchmark coverage by the number of skills included. Representative skills are extracted from large-scale task solutions on Stack Overflow using skill-aligned hierarchical clustering. Third, for real-world business tasks, we select task-solution pairs that maximize diversity in skill composition, ensuring broad coverage of practical scenarios. Fourth, to generate realistic tasks for devise domains without real tasks, we propose a systematic LLM-based task generation approach to create workflows and tasks based on these skills. Finally, we evaluate state-of-the-art data agents using our annotated benchmark and open-sourced testbed, providing detailed skill-level insights.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at how we test artificial intelligence systems designed to perform data science — and why the way we currently test them might be holding the whole field back.
Sam: So we're talking about AI agents that write code to analyze data. What's the central problem?
Alex: The core issue is that current tests only tell us whether an agent succeeded or failed a task. They don't explain why it failed — which makes it nearly impossible to fix specific weaknesses. You're essentially looking at the final exam score with no breakdown of which questions went wrong.
Sam: So we see the result, but not the reasoning behind it.
Alex: Exactly. Think about a student who keeps failing math tests. If all you know is that they failed, you can't help them. But if you know they're fine with addition and multiplication, and consistently struggle with fractions — now you can actually teach them something. To address this, the researchers built a framework called AgenticDataBench. It treats data science not as one mysterious black box, but as a collection of specific, measurable skills — each one testable on its own.
Sam: How do they actually break something as complex as a data science workflow into those measurable pieces?
Alex: They use what they call hierarchical skill extraction. Think of it like a music theory exam. Instead of just grading whether a student played a song correctly at the end, the examiner breaks the performance into individual components — scales, rhythm, dynamics — and scores each one separately. The researchers do the same thing with data science code, using an algorithm that sorts and groups coding tasks into a structured hierarchy of 433 distinct skills.
Sam: So instead of one grade, you get a detailed report card.
Alex: Exactly. And that report card tells developers something specific — whether an agent failed because it couldn't handle missing data, or because it struggled with statistical modeling. Those are very different problems requiring very different fixes.
Sam: But where do the actual test tasks come from? If the researchers are just inventing them, they might miss the messy reality of real business data.
Alex: That's a fair concern, and the authors took it seriously. They curated tasks from fifteen different domains, including real-world financial data, and used a careful selection process to ensure those tasks collectively cover the full range of skills — so nothing goes untested. For domains where real-world data is scarce, they use a pipeline that generates new tasks by sampling realistic combinations of skills. The result is a benchmark that's both broad in coverage and genuinely difficult.
Sam: So they're not just throwing random problems at the agent. They're deliberately choosing tasks to probe every corner of that skill map.
Alex: Right. And the grading works the same way — deliberately, not subjectively. Rather than asking one AI model to judge another, the system relies on execution feedback. If the agent's code actually runs and produces the correct result, it scores well. If the code crashes, times out, or returns wrong answers, the score drops. The outcome of the code running against real data is what matters — not whether the code looks good to another model.
Sam: So it's grounded in something concrete — the code either works or it doesn't.
Alex: Precisely. They also filter out skills that appear too rarely in the data, so every skill being tested has enough examples to produce a statistically meaningful result rather than a one-off fluke.
Sam: And what did all this actually reveal? Where do these agents tend to struggle?
Alex: This is where the findings get interesting. You might expect agents to stumble most on the complex modeling work — building predictive models, that sort of thing. But the largest share of failures actually came from what the paper calls "Data Analysis" tasks. Things like calculating basic statistics or summarising a dataset.
Sam: That's unexpected. Why is the analysis phase causing so many problems?
Alex: The paper suggests it comes down to a mismatch between the agent's logic and the environment it's operating in. Agents frequently hit time limits when working with large files, or they encounter a small syntax error — a missing comma, a misplaced bracket — and instead of correcting it and moving on, they get stuck in unproductive loops.
Sam: So it's not that the AI lacks the intelligence to do the analysis. It's that it gets derailed by friction — file sizes, small errors, system constraints.
Alex: That's the key insight. And notably, simply giving the agents more time to work didn't solve the problem. It just led to longer loops that still went nowhere. The bottleneck isn't raw capability — it's the interaction between the agent's decision-making and the system it's running inside.
Sam: So the implication is that we need to stop thinking of these agents as self-contained brains and start thinking of them as systems — where the environment matters just as much as the model.
Alex: That's the core takeaway. By identifying these specific failure modes, researchers can now build agents designed to handle these operational pitfalls — rather than just hoping a more powerful model will somehow figure it out on its own. The shift from a simple pass-or-fail score to a detailed skill map is what makes that kind of targeted improvement possible.
Sam: It's a much more rigorous way to hold these systems accountable — and to actually make them better.
Alex: It is. Diagnostic evaluation gives developers a map of where to look, rather than leaving them to guess. Thanks for listening to ResearchPod.