PathReportEval: A Systematic Benchmark for Pathology Report Generation

Research Motivation

Automated pathology report generation from whole-slide images (WSIs) is a critical multimodal task, yet progress is hindered by inconsistent experimental protocols and a reliance on lexical similarity metrics (e.g., BLEU, ROUGE). These metrics prioritize word overlap, often failing to identify clinically dangerous errors such as hallucinated diagnoses or discordant tumor grades. This paper addresses these issues by providing a unified benchmarking environment and a clinically grounded evaluation metric.

The PathReportEval Framework

The authors present a modular, plug-and-play framework that standardizes the entire experimental pipeline—including preprocessing, feature extraction, and model training—across three diverse datasets (TCGA, HistAI, and REG 2025). By evaluating four representative report generation models (WSI-Caption, HistGen, BiGen, and SCOUT) using three state-of-the-art pathology foundation encoders (CONCHv1.5, UNI2-h, and H-Optimus-1), the benchmark enables a fair, reproducible comparison of how different architectures and visual representations impact clinical output.

Clinical Report Quality Score (CRQS)

A central contribution is the CRQS, which moves beyond surface-level text matching. It maps both reference and generated reports into structured clinical attributes (e.g., diagnosis, grade, stage, margin status) and evaluates them across four dimensions: clinical fact coverage, key information recall, hallucination rate, and clinical discordance. This approach provides interpretable sub-scores that highlight specific failure modes, such as the omission of critical diagnostic information or the presence of contradictory clinical findings.

The study demonstrates that conventional language-generation metrics are weakly aligned with clinical correctness and often overestimate model performance. By providing a standardized benchmark and a clinically grounded metric, the authors establish a rigorous foundation for future research, ensuring that improvements in model architecture translate into safer, more accurate, and more reliable clinical reporting tools.