ADAPTIVE FILTERING OF THE KV CACHE: DIAGNOSING AND CORRECTING STRUCTURAL-ROLE BIAS IN LLM INFERENCE

As Large Language Models (LLMs) are increasingly used for tasks involving structured data like nested JSON or XML, standard KV cache compression techniques—which rely on 'attention mass' to decide which tokens to keep—often fail. This paper investigates why these methods collapse on schema-dense inputs and proposes a way to improve inference accuracy under strict memory budgets.

The author identifies a 'structural-routing bias' where attention-based eviction filters (like H2O) treat high-energy structural tokens (delimiters and keys) as more important than the actual content (values). By profiling attention mass across different token roles, the study reveals that structural tokens are over-retained, effectively pushing the relevant answer tokens out of the cache. The author introduces a role-conditional allocation method that works on top of existing windowed-attention scorers (like SnapKV) to redistribute the cache budget away from keys and toward more informative content.

• Structural Bias: In schema-dense contexts, structural tokens carry an order of magnitude more attention mass than content tokens, causing standard filters to lock onto the noise floor.
• Counterfactual Success: Simply banning the retention of 'KEY' tokens significantly improves exact-match accuracy, proving that the standard scoring mechanism is fundamentally misaligned with information relevance.
• Super-additive Gains: By combining windowed-attention scoring with a role-conditional allocator, the proposed method recovers 63–98% of the accuracy gap compared to full-cache performance at low budgets (sub-20%).
• Model Generalization: The core diagnosis holds across multiple model families (Llama, Mistral, Phi-3, Qwen), though the specific gains depend on how the model's tokenizer and architecture interact with the structural roles.