ResearchPod Summary
LLM agents are increasingly used for customer-facing tasks like booking flights or modifying orders, which require strict adherence to company policies. While prior work treats policy adherence as a safeguarding problem—blocking non-compliant tool calls based on arguments—the authors argue this is insufficient. Many policy requirements are procedural, such as verifying identity, offering insurance, or obtaining explicit user confirmation. These failures occur across multiple turns of dialogue, rendering argument-level guards blind to the context. The authors ask: can a dialogue-grounded sub-agent verifier improve policy adherence by reasoning over the full conversation and providing actionable feedback?
To address this, the authors introduce POLICYGUARD, a sub-agent verifier that sits between the agent and the environment. Unlike existing guards that only inspect tool arguments, POLICYGUARD reads the full agent-user dialogue. It uses a two-part policy specification: the raw policy document and an LLM-generated per-tool checklist. When a mutating tool call is made, the verifier evaluates it against these requirements. If the call is non-compliant, it returns a conversation-specific remediation message (e.g., 'offer insurance, then ask the user to confirm') rather than a static error. This allows the agent to correct its behavior in the same turn.
Across three frontier LLM agents (GPT-5.4, Claude Sonnet 4.6, and Gemini 2.5 Pro) on the tau^2-BENCH airline domain, POLICYGUARD consistently improved reliability. It was the only configuration to lift performance on all three agents without regressing on either refusal-required or mutation-required tasks. Specifically, it achieved perfect policy-violation recall on every agent tested. Per-call analysis revealed that POLICYGUARD is more selective than argument-level guards, blocking roughly half as often while successfully identifying more violations. Ablation studies confirmed that dialogue grounding is the primary driver of this performance, as stripping the conversation history from the verifier's input caused mutation success rates to collapse.
This paper shifts the focus of agent safety from static, argument-based filtering to dynamic, dialogue-aware verification. By treating policy adherence as a procedural workflow problem, the authors demonstrate that agents can be made more reliable without sacrificing utility. This approach provides a scalable way to enforce complex, multi-turn rules in real-world agentic applications.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.