AI Guardrail Survival under Single-Cycle Agentic Self-Summarization

Long-running AI agents periodically compact their conversation transcripts into model-generated summaries to stay within context windows. While prior work shows that dropping safety rules during compaction causes severe behavioral failures, this paper investigates the finer-grained question of how safety rules are lost under a single compaction cycle and what it implies for detection. Specifically, the study addresses four core research questions: whether safety rules survive compaction or are omitted entirely; whether rules are retained at higher rates than ordinary facts of comparable prominence; whether degraded textual residues of rules still provide behavioral protection; and whether automated LLM judges can be reliably used to evaluate post-compaction safety.

The authors utilize a rigorous experimental rig featuring a single-rule stress probe and a between-items deontic-versus-epistemic probe. Rather than relying solely on automated text analysis, the methodology incorporates a multi-tiered validation approach. First, summaries are filtered using an automated LLM judge. Crucially, these judge labels are cross-validated through blind author adjudication and behavioral replay—a process that loads summaries into a fresh agent context to test whether the surviving rules actually trigger refusals against prohibited actions. This combination of structural text analysis and functional behavioral replay isolates the specific mechanisms of memory loss during context compaction.

The central discovery is that textual presence does not equal behavioral protection. When a safety rule survives compaction only as a degraded residue rather than an intact rule, the model performs prohibited actions significantly more often (yielding behavioral gaps of +34 and +57 points across different replay models). Furthermore, rule-form items are retained substantially more often than prominence-matched facts, which creates a false sense of security because auditors assume textual survival guarantees safety. The textual form of single-cycle loss is regime-dependent, moving from a binary weld-or-drop mode under single-rule settings to degraded predicate-loss residues under tighter budget constraints. Finally, the authors document critical evaluation pitfalls where taking LLM-judge labels at face value would have reversed their scientific conclusions.