Practical Online KV Cache Compaction for LLM Agents: An Empirical Study

LLM agents operate by iteratively generating reasoning steps, calling tools, and processing environment feedback. This creates a growing KV cache that becomes a significant inference bottleneck. Unlike static document summarization, where the full context is known beforehand, agents must compress information online—often before the relevance of that information to future steps is known. This paper investigates how to adapt KV cache compaction methods to this dynamic, resource-constrained environment.

The authors evaluate two primary compaction strategies: Token Eviction (TE), which selects and retains the most important original tokens, and Attention Matching (AM), which selects tokens and then optimizes an additive bias and value representation to better approximate the original attention behavior. Because these methods rely on 'proxy queries' to determine which information to preserve, the authors test several sources: immediate boundary tokens, repeat-prefill sequences, and delayed future-turn queries generated by the agent itself.

The empirical results reveal that immediate compaction—relying on signals available at the end of the current turn—often degrades performance. By delaying compaction by even a single turn, the agent can use its own subsequent generation as a proxy, which significantly improves the quality of the compressed cache. While AM is theoretically more sophisticated, TE proves surprisingly robust to imperfect proxy signals, making it a more practical choice for many agentic workflows. Across different model scales, TE successfully reduced the KV cache footprint by up to 80% while maintaining performance levels comparable to uncompressed baselines.

As agents are deployed for increasingly long-horizon tasks, managing the KV cache becomes critical for both cost and latency. This study provides a practical roadmap for developers, demonstrating that the choice of proxy-query source is a primary design lever. It suggests that systems should prioritize simple, robust selection methods like TE and incorporate a small delay to leverage future-turn signals, rather than attempting complex, immediate optimization.