Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems

Modern agentic AI systems often employ detect-and-block mechanisms to filter malicious inputs and outputs. While these defenses are effective against isolated attacks, they create a predictable feedback loop for automated adversaries. When a defense system consistently returns a standard refusal message, it provides the attacker's automated judge with clear, binary feedback. This allows model-guided attack frameworks—such as PAIR or GPTFuzz—to iteratively refine their prompts, effectively learning the boundaries of the defense and optimizing for success over time. The authors show that as the attacker's query budget grows, the probability of a successful jailbreak approaches one, as the defense's own responses inadvertently guide the attacker toward effective adversarial prompts.

To counter this, the authors propose a shift from blocking to misdirection. Instead of returning a predictable refusal, the system provides a non-operational, contextually plausible response. This strategy aims to disrupt the attacker's feedback loop by inducing false-positive errors in the automated judge. Because the judge relies on heuristic cues—such as tone, structure, and semantic relevance—to evaluate whether an attack was successful, these misleading responses can trick the judge into incorrectly labeling a failed attack as a success. This degradation of the judge's positive predictive value prevents the attacker from reliably identifying which prompts are actually effective, thereby bounding the attacker's success rate even as the search continues.

To realize this strategy, the authors introduce Contextual Misdirection via Progressive Engagement (CMPE). CMPE replaces standard refusal text with responses that use positive-intent framing and safe, conversational engagement. By simulating a helpful but ultimately non-operational interaction, CMPE exploits the limitations of current LLM-based judges that prioritize surface-level coherence over deep semantic verification. In empirical evaluations against PAIR and GPTFuzz, CMPE reduced estimated attack success rate (ASR) upper bounds by up to two orders of magnitude and nearly eliminated verified attack success in end-to-end testing.