ResearchPod Summary
BankerToolBench (BTB) is an open-source benchmark designed to test AI agents on realistic, end-to-end workflows from investment banking (IB)—the high-stakes world of financial analysis, modeling, and client pitches. Unlike toy tasks or isolated skills, BTB simulates what junior bankers actually do: spending up to 21 hours per task navigating data rooms, pulling market data and SEC filings via APIs, and producing multi-file deliverables like Excel financial models, PowerPoint pitch decks, and PDF/Word reports. Developed with input from 502 bankers at top firms, BTB ensures ecological validity—tasks mirror real professional environments, not academic abstractions.
Think of it as a 'gym' for AI agents in finance: senior bankers issue prompts (e.g., 'Build a DCF/SOTP valuation model for Capricor Therapeutics'), and agents must use industry tools to deliver client-ready outputs. This contrasts with prior benchmarks like APEX-Agents or GDPVal, which oversimplify IB by ignoring multi-file coordination, incomplete rubrics, or junior-banker realities.
BTB's killer feature is its deliverable-centric assessment via an Agentic Verifier—an automated grader scoring 100+ criteria crafted by veteran bankers. Rubrics cover instruction following (e.g., toggle switches in Excel loaders), financial accuracy (e.g., linked revenues, not hardcoded), cross-artifact consistency (e.g., pitch deck matching model outputs), and client readiness (e.g., polished visuals, error-free logic).
No more subjective human grading or simple text pass/fail: the verifier handles complex artifacts like spreadsheets, enabling reproducible, scalable eval. Bankers rate outputs on utility—0% of top model outputs were client-ready, highlighting the gap.
Testing 9 leading models (e.g., GPT-5.4 as top performer), BTB reveals stark limitations:
This underscores AI's shift needed from discrete tasks (text drafting, data lookup) to agentic systems for economic value in finance, where reliability drives adoption (like high-frequency trading).
Alex: Welcome to another episode of ResearchPod.
Sam: Today, we're looking at a research paper called BankerToolBench, or BTB. It creates a new test for how well top AI systems handle full work processes in investment banking—like advising on mergers or raising money through stocks and bonds.
Alex: So this is a realistic test of AI doing a junior banker's complete job?
Sam: Yes. Junior bankers spend five to 21 hours on tasks like pulling market data and official company reports, then building linked Excel sheets for cash flow predictions. The benchmark has 100 such tasks, based on input from over 500 real bankers. It mimics the full chain of steps, ending in client-ready files like Excel models or PowerPoint decks. Even the best AI, like GPT-5.4, fails nearly half the detailed checks and gets zero percent rated as usable without fixes.
Alex: What makes these tasks so tough?
Sam: These are workflows where one small mistake—like a wrong formula link between sheets—ruins everything. It's like gears in a machine: if one slips, the whole thing grinds to a halt. Bankers need real data tools, judgment on assumptions, and consistency across files. AI struggles with that in long chains, especially coordinating between files.
Alex: How did researchers make sure the tasks match real junior banker work?
Sam: They surveyed hundreds of bankers and interviewed seniors to map common tasks, like valuation models or pitch decks. Then, 172 experienced bankers from top firms recreated real tasks they'd done, with their bosses' prompts. Each made a perfect example file—a 'golden deliverable'—and built checklists with over 100 yes-or-no checks across six areas, like correct formulas or professional looks. Multiple bankers reviewed everything.
Alex: And how do top AIs score on that?
Sam: GPT-5.4 passes all critical checks on just 16% of tasks. Failures often hit basics, like using fixed numbers instead of live formulas that update automatically.
Alex: So these workflows show AI's weak spots in linking steps over long processes.
Alex: If models struggle with clear prompts from real bankers, does adding extra guidance—like banking know-how or layout rules—help?
Finance is a canary in the coal mine for AI labor impact: early automation adopter when tools prove reliable. BTB quantifies the chasm—top models fail half the criteria—pinpointing fixes like better multi-step reasoning, tool use, and verification. Contributions include:
Limitations acknowledged: task scope (analytical IB), potential rubric biases, compute-heavy evals. Future: expand to M&A, trading; integrate human-AI loops.
Intuition builder: Imagine delegating a full pitch to AI—BTB tests if it's ready to replace (or augment) a $200k/year junior banker, with real economic stakes.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: Bankers assume colleagues know unwritten rules, like standard sheet labels or valuation steps. The study added explanations of those details and step-by-step file notes. Scores lifted across models, suggesting AI lacks built-in job knowledge—but training on banking data could help.
Alex: Does the software setup around the AI matter?
Sam: The software wrapper—or agent harness—lets AI call data, edit files, and manage conversations. Researchers tested three open-source ones with strong models. Differences were small, but runtimes varied—one was quicker without much accuracy loss.
Alex: Can models improve by practicing on the benchmark?
Sam: Post-training lets AI try tasks repeatedly and rewards good rubric scores—like coaching an athlete with feedback. They used two methods on smaller Qwen models, practicing on 80 tasks and testing on 20 new ones. Scores rose substantially.
Alex: The paper sounds cautious about real-world limits.
Sam: Yes—like clean data instead of messy real deals, no team collaboration, and a US focus. Still, it shows AI handles pieces well but falters on full chains where one slip breaks everything.
Alex: What do detailed failure examples show?
Sam: One issue: AI invents fake details, like a nonexistent PowerPoint command that crashes code—then deletes the line instead of fixing it. This happens in 41% of coding failures; AI patches the surface but skips checking if it works.
Alex: Like cheating on a recipe by skipping a step?
Sam: Precisely. Another: In Excel, changing one number—like a purchase price—should update everything through linked formulas, like gears in a machine. But AI writes fixed values, ignoring real formulas, so nothing tweaks.
Alex: Do these tie to how AI uses data-fetching or coding helpers?
Sam: Tool errors are highest for code scripts at about 11%, often basic mistakes. GPT-5.4 has just 2% overall, but fabricates data when sources fail—like inventing drug trial stats.
Alex: How well do rubric scores match what bankers think?
Sam: Bankers rated AI files on readiness for bosses. A score cutoff made outputs 3.35 times more likely to pass—nearly matching banker agreement at 3.68 times. On 1,356 checks, the verifier hit 88% accuracy against human consensus.
Alex: That comes from actually running the files?
Sam: Yes—with tools like LibreOffice to recalculate formulas, checking six categories from technical correctness to client polish. It excels at formula verification but catches skipped formatting too.
Alex: Pulling it all together, what does this say about AI's readiness for junior banker shifts?
Sam: Top models pass key checks on only about one in six tasks, with zero fully client-ready. Failures cluster on basics—like fixed numbers instead of live formulas, or balance sheets that don't add up. The paper flags limits like simplified setups and no team handoffs—plus a US focus. Still, it reveals gaps in end-to-end reliability for high-stakes finance. The public toolkit could train better agents, letting AI handle routine modeling so humans focus on strategy. It's a notable step in measuring progress, though full autonomy remains distant.
Alex: That's a grounded view of where things stand. Thanks, Sam. Thanks for listening to ResearchPod.