Epiphany-Aware KV Cache Eviction Without the Attention Matrix

As reasoning models generate increasingly long chains of thought, the Key-Value (KV) cache grows linearly, creating a significant memory bottleneck during deployment. Traditional cache eviction methods rely on attention weights to rank token importance, but this requires materializing the attention matrix, which is computationally expensive and prevents the use of optimized, fused kernels like FlashAttention. This paper asks whether it is possible to identify important tokens for eviction using only internal model representations, thereby bypassing the need for attention-based scoring.

The authors propose a metric called the epiphany score, which measures the L2-norm change in hidden states across specific layers during the forward pass. By analyzing the model's internal anatomy, they identify two distinct bands of layers: a positive band (layers 7–13) where hidden-state changes correlate with token importance, and a negative band (layers 18–25) where changes correlate with dispensable tokens. The final score is computed by taking the difference between the rolling z-scores of these two bands. This method is fully causal, requires no training, and is compatible with standard FlashAttention inference stacks.

EpiKV demonstrates that attention-based scoring is not strictly necessary for effective cache eviction. On the MATH-500 benchmark, EpiKV achieves 72% accuracy at a 4096-token budget, matching or exceeding existing attention-based baselines. Furthermore, because the method avoids materializing the attention matrix, it allows for significantly longer context windows—up to 16x longer than eager-attention methods—and provides a speedup of up to 2.8x in wall-clock time. The authors also show that the layer-based signal is robust, though it adapts to the difficulty of the task, suggesting it captures the actual location of load-bearing computation within the model.