ResearchPod Summary
Agent evaluations and trace-based learning pipelines frequently compare model outputs across transformed views, such as modified prompts, tool schemas, or repositories, by applying a post-response correspondence step. This paper investigates whether this correspondence is truly neutral preprocessing or an active measurement intervention. The authors develop a formal validity theory and audit framework comprising three components: two-sided validation of nuisance removal and response preservation, all-optima identification of downstream conclusions, and uncertainty propagation across exact-optimal correspondence sets. To test these principles, the study examines public code and SQL pipelines, conducts frozen tool-use audits including a task-and-seed-disjoint replication, and evaluates transport gates on natural and corrected responses.
The central claim of the paper is that a point conclusion is only solver-independent if the correspondence rule is declared, validated for both nuisance removal and response preservation, and supported by every legal exact optimum. The authors show that benign-only calibration is insufficient because it cannot prevent false invariance; a transport map calibrated solely on benign examples can inadvertently erase response-bearing differences. The proposed two-sided alignment certificate enforces bounds on null leakage and restricted response gain simultaneously. Furthermore, the authors define a correspondence contract that pairs a correspondence objective with a downstream readout policy, ensuring that downstream analyses do not rely on an arbitrary solver tie-break.
Validating the transport does not automatically resolve ties when multiple exact-optimal matchings exist. To handle this, the authors introduce the legal update body, which is the convex hull of compiled updates across all exact optima. By analyzing the geometry of these update bodies, the study establishes an update trichotomy that determines whether an update is solver-identified, achieves positive margin, or reaches the origin. The complexity analysis reveals a sharp tractability frontier: while affine compilers and local nonlinear compilers with low treewidth permit efficient exact auditing, shared normalization compilers transform standardization into an Ising-like problem, making universal all-optima threshold certification NP-hard and coNP-hard.
Natural audits reveal substantial hidden ambiguity in existing evaluation pipelines. Across public SWE-bench and Spider2-DBT trajectory pairs, two deterministic optimal tracebacks disagree on temporal localization for over 55 percent of nonzero pairs. Frozen tool-use audits further expose exact-optimum reversals of intended turn-level credit. Additionally, a pre-registered transport gate failed on natural responses, and held-out controls demonstrate that maps calibrated exclusively on benign examples erase every retained harmful response, whereas two-sided validation successfully selects response-preserving alternatives. Consequently, cross-view correspondence must be explicitly declared, validated, and propagated into uncertainty before drawing point conclusions.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.