Prior Audit–Repair Context Shifts LLM Verifier Thresholds Toward Leniency

Automated checking pipelines frequently use language models in sequence, where one model checks a reasoning trace (the checker) and another repairs what is flagged (the fixer). This paper investigates whether placing a completed audit-repair episode in a model's context alters what the checker reports on subsequent tasks. Specifically, the authors measure the false-alarm rate (FAR) on human-verified-correct ProcessBench traces where the task is held byte-identical.

To separate context effects from audit-specific effects, the study employs multiple controls—including length-matched non-audit fillers, inert continuations, and incorrect-verdict episodes—across three model families and five semantically matched prompt wordings. Because a lower false alarm rate could simply indicate a general reluctance to flag errors, the authors pair the clean target arm with a second, disjoint arm of labelled-incorrect traces to perform signal-detection analysis, calculating both discrimination and decision criteria.

A prior audit-repair episode reliably lowers false alarms in 15 out of 15 model and wording combinations, reducing FAR by 2.8 to 11.5 percentage points compared to length-matched non-audit controls. Counter to what accumulated-message literature and negativity asymmetries predict, episodes featuring an audit that reported an error actually lower false alarms further still, demonstrating that polarity drift alone does not explain the phenomenon. Signal-detection analysis confirms this shift is driven by a change in the decision threshold rather than improved discrimination. Finally, a hand audit of the removed false alarms reveals that 82% were simply wrong, indicating that this leniency shift is not harmful at this operating point.

These findings serve as a methodological caution for automated verification pipelines: engineers must be wary of placing verifiers in contexts where they have recently performed similar repairs. Because standard metrics computed solely on flagged items will interpret this threshold shift as an improvement in accuracy, pipeline designers risk misinterpreting criterion leniency as genuine capability gains.