MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

Large language model (LLM) agents generate complex, heterogeneous execution traces that include system instructions, plans, tool outputs, and intermediate reasoning. As these traces grow, the key-value (KV) cache becomes a significant memory bottleneck. Existing eviction policies often treat all tokens equally, ignoring the semantic structure of the agent's prompt. The authors propose MemDecay, a training-free framework that assigns tokens region-specific priorities and decay rates. By combining these structural priors with attention-derived importance, MemDecay manages cache memory at a page-granular level, allowing for the protection of critical regions while evicting less relevant tokens.

MemDecay demonstrates that semantic prompt structure is a robust signal for KV-cache management. In experiments across different context lengths and model sizes, the authors found that attention lifetimes vary by an order of magnitude across different regions—for instance, system tokens have significantly longer half-lives than scratchpad reasoning tokens. Pinning system-region facts ensures they remain in the cache at full accuracy, whereas standard recency-based policies fail as context grows. While MemDecay's structural approach is highly effective, the authors identify that the policy's attention-score normalization is a limitation, as it currently struggles to fully compensate for unpinned content compared to pure accumulated-attention baselines.

As LLM agents are increasingly deployed for long-horizon tasks, efficient memory management is essential for scalability. MemDecay provides a transparent, interpretable, and training-free method to optimize KV cache usage without requiring auxiliary models or complex retraining. By aligning cache eviction with the functional roles of prompt segments, this approach offers a practical path toward more reliable and memory-efficient agentic workflows.