Handshake AI, Elaine Lau, Markus Dücker, Ronak Chaudhary, Hui Wen Goh, Rosemary Wei, Vaibhav Kumar, Saed Qunbar, Guram Gogia, Yi Liu, Scott Millslagle, Nasim Borazjanizadeh, Ulyana Tkachenko, Samuel Eshun Danquah, Collin Schweiker, Vijay Karumathil, Asrith Devalaraju, Varsha Sandadi, Haemi Nam, Punit Arani, Ray Epps, Abdullah Arif, Sahil Bhaiwala, Curtis Northcutt, Skyler Wang, Anish Athalye, Jonas Mueller, Francisco Guzmán
6 min
BankerToolBench (BTB) is an open-source benchmark designed to test AI agents on realistic, end-to-end workflows from investment banking (IB)—the high-stakes world of financial analysis, modeling, and client pitches. Unlike toy tasks or isolated skills, BTB simulates what junior bankers actually do: spending up to 21 hours per task navigating data rooms, pulling market data and SEC filings via APIs, and producing multi-file deliverables like Excel financial models, PowerPoint pitch decks, and PDF/Word reports. Developed with input from 502 bankers at top firms, BTB ensures ecological validity—tasks mirror real professional environments, not academic abstractions.
Think of it as a 'gym' for AI agents in finance: senior bankers issue prompts (e.g., 'Build a DCF/SOTP valuation model for Capricor Therapeutics'), and agents must use industry tools to deliver client-ready outputs. This contrasts with prior benchmarks like APEX-Agents or GDPVal, which oversimplify IB by ignoring multi-file coordination, incomplete rubrics, or junior-banker realities.
BTB's killer feature is its deliverable-centric assessment via an Agentic Verifier—an automated grader scoring 100+ criteria crafted by veteran bankers. Rubrics cover instruction following (e.g., toggle switches in Excel loaders), financial accuracy (e.g., linked revenues, not hardcoded), cross-artifact consistency (e.g., pitch deck matching model outputs), and client readiness (e.g., polished visuals, error-free logic).
No more subjective human grading or simple text pass/fail: the verifier handles complex artifacts like spreadsheets, enabling reproducible, scalable eval. Bankers rate outputs on utility—0% of top model outputs were client-ready, highlighting the gap.
Testing 9 leading models (e.g., GPT-5.4 as top performer), BTB reveals stark limitations:
This underscores AI's shift needed from discrete tasks (text drafting, data lookup) to agentic systems for economic value in finance, where reliability drives adoption (like high-frequency trading).
Finance is a canary in the coal mine for AI labor impact: early automation adopter when tools prove reliable. BTB quantifies the chasm—top models fail half the criteria—pinpointing fixes like better multi-step reasoning, tool use, and verification. Contributions include:
Limitations acknowledged: task scope (analytical IB), potential rubric biases, compute-heavy evals. Future: expand to M&A, trading; integrate human-AI loops.
Intuition builder: Imagine delegating a full pitch to AI—BTB tests if it's ready to replace (or augment) a $200k/year junior banker, with real economic stakes.
Existing AI benchmarks lack the fidelity to assess economically meaningful progress on professional workflows. To evaluate frontier AI agents in a high-value, labor-intensive profession, we introduce BankerToolBench (BTB): an open-source benchmark of end-to-end analytical workflows routinely performed by junior investment bankers. To develop an ecologically valid benchmark grounded in representative work environments, we collaborated with 502 investment bankers from leading firms. BTB requires agents to execute senior banker requests by navigating data rooms, using industry tools (market data platform, SEC filings database), and generating multi-file deliverables--including Excel financial models, PowerPoint pitch decks, and PDF/Word reports. Completing a BTB task takes bankers up to 21 hours, underscoring the economic stakes of successfully delegating this work to AI. BTB enables automated evaluation of any LLM or agent, scoring deliverables against 100+ rubric criteria defined by veteran investment bankers to capture stakeholder utility. Testing 9 frontier models, we find that even the best-performing model (GPT-5.4) fails nearly half of the rubric criteria and bankers rate 0% of its outputs as client-ready. Our failure analysis reveals key obstacles (such as breakdowns in cross-artifact consistency) and improvement directions for agentic AI in high-stakes professional workflows.
Sam: The software wrapper—or agent harness—lets AI call data, edit files, and manage conversations. Researchers tested three open-source ones with strong models. Differences were small, but runtimes varied—one was quicker without much accuracy loss.
Alex: Can models improve by practicing on the benchmark?
Sam: Post-training lets AI try tasks repeatedly and rewards good rubric scores—like coaching an athlete with feedback. They used two methods on smaller Qwen models, practicing on 80 tasks and testing on 20 new ones. Scores rose substantially.
Alex: The paper sounds cautious about real-world limits.
Sam: Yes—like clean data instead of messy real deals, no team collaboration, and a US focus. Still, it shows AI handles pieces well but falters on full chains where one slip breaks everything.
Alex: What do detailed failure examples show?
Sam: One issue: AI invents fake details, like a nonexistent PowerPoint command that crashes code—then deletes the line instead of fixing it. This happens in 41% of coding failures; AI patches the surface but skips checking if it works.
Alex: Like cheating on a recipe by skipping a step?
Sam: Precisely. Another: In Excel, changing one number—like a purchase price—should update everything through linked formulas, like gears in a machine. But AI writes fixed values, ignoring real formulas, so nothing tweaks.
Alex: Do these tie to how AI uses data-fetching or coding helpers?
Sam: Tool errors are highest for code scripts at about 11%, often basic mistakes. GPT-5.4 has just 2% overall, but fabricates data when sources fail—like inventing drug trial stats.
Alex: How well do rubric scores match what bankers think?
Sam: Bankers rated AI files on readiness for bosses. A score cutoff made outputs 3.35 times more likely to pass—nearly matching banker agreement at 3.68 times. On 1,356 checks, the verifier hit 88% accuracy against human consensus.
Alex: That comes from actually running the files?
Sam: Yes—with tools like LibreOffice to recalculate formulas, checking six categories from technical correctness to client polish. It excels at formula verification but catches skipped formatting too.
Alex: Pulling it all together, what does this say about AI's readiness for junior banker shifts?
Sam: Top models pass key checks on only about one in six tasks, with zero fully client-ready. Failures cluster on basics—like fixed numbers instead of live formulas, or balance sheets that don't add up. The paper flags limits like simplified setups and no team handoffs—plus a US focus. Still, it reveals gaps in end-to-end reliability for high-stakes finance. The public toolkit could train better agents, letting AI handle routine modeling so humans focus on strategy. It's a notable step in measuring progress, though full autonomy remains distant.
Alex: That's a grounded view of where things stand. Thanks, Sam. Thanks for listening to ResearchPod.