VETSCORE: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations

Research Question and Approach

In high-stakes domains like veterinary medicine, Large Language Models (LLMs) often generate text that is not fully supported by the citations they provide. Existing fact-verification methods, such as FactScore, typically treat all claims as equally important. However, in a clinical setting, an unverified claim about a drug dosage is far more dangerous than an unverified claim about the year a study was published. The authors introduce VetScore, a pipeline designed to address this by decomposing model outputs into atomic claims, assessing each claim's potential for clinical harm, and verifying its faithfulness to provided source excerpts.

The VetScore Pipeline

The VetScore process consists of four main steps:

1. Segmentation: The output is split into segments based on citations and formatting.
2. Claim Decomposition: Each segment is broken down into atomic claims, using previous context to resolve anaphora.
3. Harm Potential Scoring: Each claim is assigned a score from 1 (no clinical consequence) to 5 (direct patient harm) by an LLM.
4. Fact Verification: Each claim is checked against the provided source excerpts to determine if it is supported.

These components are aggregated into a final risk-adjusted score, which represents the fraction of total potential harm that remains unsupported by the citations. This allows practitioners to identify high-risk, unverified information quickly.

By incorporating a risk-weighting mechanism, VetScore provides a more nuanced and clinically useful evaluation of AI-generated medical content. Unlike standard metrics that treat all errors as equal, this approach prioritizes the detection of high-stakes inaccuracies. The authors demonstrate that even smaller judge models can achieve high correlations with human veterinary experts, making this an efficient tool for real-time validation and improving the reliability of evidence-based veterinary QA systems.