PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noise

Large Language Models (LLMs) are increasingly integrated into decision-support systems for high-stakes fields like education and healthcare. However, most existing benchmarks assume that tools provide perfectly accurate information. In reality, these tools are often noisy, and human decision-makers must navigate this uncertainty. This paper introduces PREDACTBENCH, a benchmark designed to evaluate how well LLM-based agents and their human collaborators perform when paired with statistically imperfect, noisy tools.

PREDACTBENCH focuses on academic risk prediction, where an LLM assistant uses a k-nearest neighbor (k-NN) predictor to analyze student performance and suggest interventions. The benchmark introduces controlled noise into the predictor, varying its accuracy from 40% to 80%. It evaluates performance using two key metrics: Relative AI-Reliance (RAIR) and Relative Self-Reliance (RSR). These metrics extend trust calibration frameworks to multi-turn dialogues, measuring whether an agent or human correctly updates their decision when the AI is right (RAIR) and whether they correctly ignore the AI when it is wrong (RSR).

The researchers evaluated 13 state-of-the-art LLMs and conducted a human study with instructors and teaching assistants. The results reveal a significant gap in performance: while human participants demonstrated better-calibrated reliance—effectively distinguishing between helpful and misleading AI advice—most LLMs failed to provide the necessary visibility to avoid over-relying on incorrect suggestions. Even the most advanced models struggled to achieve the balance of trust observed in human decision-makers, highlighting a critical limitation in current AI decision-support systems.

As LLMs are deployed to assist in high-stakes decision-making, the ability to handle tool uncertainty is paramount. This study demonstrates that current models are prone to "blind" trust in their own tool-augmented outputs. By providing a standardized way to measure trust calibration, PREDACTBENCH offers a path toward developing more robust, transparent, and reliable AI agents that can better support human experts.