ResearchPod Summary
Deploying LLM agents in software systems carries significant risk, as agents may accidentally delete data, misuse tools, or misconfigure critical records. Existing benchmarks often lack construct validity, failing to reflect the specific workflows and environments where agents are actually deployed. While some developers use replica environments to test agents, these are expensive to maintain and prone to configuration drift, making them unsuitable for rapid iteration.
This paper introduces Copy-on-Write (CoW) Scoring, a framework that enables safe, application-specific agent evaluation. By leveraging a PostgreSQL-level mechanism, the framework splits database tables into a base table (containing production data) and a changes table (capturing agent-specific writes). A database view merges these tables, allowing agents to interact with the system as if they were operating on live data, while all modifications are isolated to the changes table. This allows for inexpensive, repeatable testing without risking production integrity.
The framework evaluates agents by comparing their session outputs against a human-generated ground-truth (GT) session. It provides two levels of granularity:
The authors demonstrated CoW Scoring on Plane, an open-source project-management platform. By analyzing agent performance, they identified specific failure modes, such as vocabulary mismatches (e.g., the agent searching for "issues" when the system used "work items") and hallucinations of invalid parameters. After updating the tool surface to address these issues, the researchers observed measurable performance improvements across multiple models. This approach provides a repeatable "score-diagnose-fix" loop that can be integrated into any application to improve agent reliability.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.