FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

Deep research agents are increasingly used to generate complex financial reports, yet evaluating these systems remains a bottleneck. Traditional evaluation often requires human experts to design and execute high-quality rubrics, which is costly and difficult to scale. This paper addresses this challenge by proposing an automated pipeline that generates query-specific rubrics and filters them to create a robust, expert-free benchmark.

The authors collected 104 real-world financial queries and 1,040 corresponding reports to form a benchmark dataset. They then synthesized 14,450 candidate rubric items—binary criteria used to assess report quality. To remove the need for human experts in the final execution loop, the researchers validated a three-LLM judge panel against human experts. They found that LLM unanimity is a strong proxy for human agreement, achieving 98.67% label-level agreement on jointly unanimous items.

To ensure the rubrics are both reliable and useful for ranking, the authors applied two filters to the candidate pool:

1. Strict Consistency Filter: Retains only rubrics where all three LLM judges unanimously agree on every report for a given query.
2. Distinguishability Filter: Retains only rubrics that assign both 'yes' and 'no' labels across the evaluated systems, ensuring the criteria can actually differentiate between high- and low-performing reports.

This process reduced the 14,450 candidates to 2,600 consensus-derived gold rubrics. When applied to 10 deep research systems, these rubrics produced clear, stable rankings with pass-rate gaps exceeding 36 percentage points, demonstrating that the pipeline effectively scales evaluation without sacrificing discriminative power.

By removing the need for human experts in the final evaluation loop, this pipeline provides a scalable framework for benchmarking and iterative system improvement. It demonstrates that high-quality, domain-specific evaluation can be automated by leveraging consensus among multiple LLM judges, offering a path forward for evaluating long-form, data-dense financial reports at scale.