ResearchPod Summary
Long-context language models rely on the KV-cache to maintain state during autoregressive decoding. While compressing this cache is essential for efficiency, aggregate performance metrics often mask why specific, previously correct outputs fail after compression. The authors introduce KVDiagnosis, a diagnostic benchmark designed to move beyond aggregate scores by providing a systematic, per-source analysis of compression-induced failures.
The researchers developed a taxonomy of 25 compression methods, grouping them into five mechanism families (e.g., token eviction, budget allocation, channel pruning). They verified eight implementations and established a rigorous evaluation pipeline: every source is first evaluated with a FullCache control, and only then are 'Correct-to-Wrong' (C-to-W) failures identified for each specific method-setting. This ensures that failure sets are not biased by the definitions of other compressors.
Using the Qwen3-8B model across four evidence-aware workloads, the study generated 59,800 compressed runs and 12,520 C-to-W failure rows. The analysis revealed that 63.2% of these failures suffer from low or partial coverage of the necessary evidence tokens. Interestingly, only 0.2% of failures combine high evidence coverage with strong likelihood drift, suggesting that most failures are not simply due to minor numerical approximations but rather fundamental losses of critical information.
The study also demonstrates that different compression methods, even when achieving similar aggregate task scores, often fail on entirely different sets of source inputs. Furthermore, the authors validated their diagnostic framework by showing that a targeted 4x boost in evidence-attention could repair 29.2% of low-evidence-attention-retention (low-EAR) failures, significantly outperforming sham interventions.
This work provides a critical tool for researchers to move from 'black-box' evaluation to 'white-box' debugging of long-context models. By linking specific failure modes to cache, likelihood, and attention traces, KVDiagnosis allows developers to identify whether a failure is caused by the loss of critical evidence, representation degradation, or decoding errors. This distinction is vital for selecting the right repair strategy—such as increasing precision versus adjusting eviction policies—and for understanding the true trade-offs between memory efficiency and model reliability.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.