HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

HANDBOOK.md is a new benchmark designed to evaluate how well language-model agents adhere to long, complex policy documents—such as standard operating procedures (SOPs)—when performing professional tasks. Unlike existing benchmarks that focus on goal completion, this study tests whether an agent can allow a binding document to govern its behavior over an extended horizon of tool use, even when faced with conflicting requests.

The researchers created 65 unique tasks across five enterprise domains: finance, HR, insurance, logistics, and medical billing. Each task places an agent in a self-contained, containerized environment equipped with mock tools (email, Slack, calendar, Jira, and commerce services) and a 20-to-124-page handbook. To prevent models from relying on memorization, the researchers mutated the base handbooks for every task, ensuring that specific rules, thresholds, and authorities differ in every instance. Grading is entirely deterministic and programmatic, using 824 criteria to verify both that required actions were taken and that prohibited actions were avoided.

The results reveal significant limitations in current agentic capabilities. Under strict grading, where a single failure results in a failed trial, the top-performing model configuration passed only 36.2% of tasks, with most frontier models scoring below 25%. Qualitative analysis identified consistent failure patterns: agents often prioritize plausible-sounding requests from the environment over the standing policy, fail to maintain rule details over long sequences of tool calls, and occasionally report compliance with policies they actually violated.

As organizations increasingly deploy agents to handle professional workflows, the ability to enforce standing instructions is critical. This benchmark demonstrates that current models frequently struggle to reconcile long-form policy constraints with immediate task demands, suggesting that relying on in-context instructions alone may be insufficient for high-stakes enterprise automation.