VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference

As Speech Large Language Models (SLLMs) scale to handle long-form audio tasks like meeting summarization and multi-turn dialogue, the memory overhead of the Key-Value (KV) cache becomes a critical bottleneck. Existing compression methods designed for text or vision often fail to account for the unique temporal redundancy and low information density of audio, leading to semantic drift or the loss of crucial paralinguistic cues. VoxZip aims to solve this by efficiently compressing the KV cache without sacrificing the model's ability to reason over long audio contexts.

VoxZip introduces a two-stage, train-free compression framework:

1. Semantic-Anchored Compression (Prefill Stage): The model uses an auxiliary ASR system to transcribe the audio. These transcriptions serve as "semantic anchors." The framework aligns raw audio tokens with these anchors, averaging audio features within each anchor's temporal window and fusing them with the corresponding text embeddings. This increases information density and reduces the initial cache size.

2. Temporally Decayed Eviction (Decoding Stage): During autoregressive generation, the model must decide which tokens to keep in the cache. Standard attention-based pruning often suffers from "accumulation bias," where early tokens are unfairly favored. VoxZip introduces a temporal decay factor to the accumulated attention scores, ensuring that the model prioritizes currently relevant information while pruning stale or redundant acoustic tokens.

VoxZip demonstrates superior performance across six audio benchmarks, including long-context reasoning tasks like Vox-Infinity and general audio understanding tasks. At a 4x compression ratio, the system achieves a 1.9x increase in inference throughput and a 3.3x reduction in peak memory usage. Even under an aggressive 20x compression, the model retains over 90% of its uncompressed baseline performance, proving that the semantic-anchored approach effectively preserves both linguistic and paralinguistic information.