Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents

The Shift to Out-of-Band Defense

Recent research in securing LLM agents has moved away from trying to train models to refuse malicious instructions (in-band defense). Instead, the field has converged on out-of-band enforcement, where a deterministic policy layer sits outside the model to mediate its actions. Systems like CaMeL, FIDES, and Progent use techniques such as capability-based security, information-flow labels, and reference monitors to ensure that even if an agent is tricked by an indirect prompt injection, it cannot perform unauthorized actions like deleting files or transferring funds.

The Evaluation Gap

While these systems report high success rates on static benchmarks, the authors argue that this is the same methodology that previously led to overconfidence in in-band defenses. Historically, in-band defenses appeared robust until researchers applied adaptive, defense-aware attacks, which caused success rates to plummet. The authors contend that the current generation of action-level defenses has not yet been subjected to a standardized, independent, and rigorous adaptive evaluation. Relying on static benchmarks creates a blind spot, as these tests measure resistance to fixed attack sets rather than resilience against an attacker who knows the defense and is actively optimizing against it.

Independent Reproduction and Extension

To address this, the authors performed an independent reproduction of Progent’s adaptive-attack analysis using an open-weight Qwen2.5-7B model. In their experiments, Progent reduced the mean attack success rate from 25.8% to 4.2%. When they introduced a hand-crafted adaptive attack, the success rate remained low at 2.6%. While these results suggest that deterministic enforcement is a harder target for attackers than model-based detection, the authors emphasize that this is a small-scale data point. They caution that stronger, white-box optimization attacks (such as GCG) have not yet been tested against these systems.

This paper provides a crucial reality check for the security of agentic AI. By framing current defenses through the lens of classical security principles—such as Biba integrity and the Saltzer–Schroeder principles—the authors provide a structured way to compare disparate systems. Their work highlights that while the industry has made significant progress in moving security out of the model, the field must now shift its focus toward rigorous, adaptive red-teaming to ensure these defenses hold up against sophisticated, evolving threats.