When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents

This paper investigates whether exact-match retrieval recall—the standard metric for evaluating policy retrieval in tool-use agents—is a valid proxy for how well a downstream decision model performs. The authors test whether a classifier's ability to make correct decisions (e.g., allow, verify, or refuse an action) actually depends on retrieving the exact benchmark-designated policy clause, or if "near-miss" retrieved clauses provide sufficient context for the model to succeed.

The researchers use the tau-bench airline domain to train Qwen2.5-3B and 7B classifiers. They first establish that providing a structured decision state (containing the applicable policy) significantly improves performance over raw trajectory text. They then replace the "gold" policy clause with retrieved clauses from off-the-shelf retrievers. To evaluate the validity of exact-match recall, they conduct a "gold-injection" diagnostic to determine the threshold of exact-clause access required for high performance, and then compare this to the performance of a classifier using actual retrieved clauses.

The study finds that while off-the-shelf retrievers recover the exact governing policy clause only 7% of the time, the downstream classifier performance is nearly identical to that achieved with the gold policy. The retrieved clauses, despite not being an exact match, provide enough decision-relevant information to significantly outperform mismatched-policy or no-policy controls. This suggests that exact-match recall is overly pessimistic and fails to capture the utility of non-matching but contextually relevant retrieved information. However, this result is configuration-dependent; the performance gap between gold and retrieved policies re-emerges in under-trained or overly-discriminating models.

These results demonstrate that researchers should be cautious when using retrieval recall as a standalone metric for agent performance. Because non-matching clauses can still be highly informative, optimizing solely for exact-match recall may lead to suboptimal system design. The authors argue that evaluation should shift toward measuring the downstream utility of retrieved policies within the decision-making loop rather than relying on retrieval-only metrics.