C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

Overview

Long-context LLM applications, such as retrieval-augmented generation (RAG), face significant bottlenecks due to the high cost of storing and accessing large key-value (KV) caches. While existing methods attempt to reduce recomputation through KV cache reuse, they often struggle with the storage and memory bandwidth constraints inherent in long-context serving. C2KV addresses these challenges by introducing a unified framework that jointly optimizes KV cache compression and concatenation, enabling efficient, modular reuse of compressed KV representations.

The C2KV Approach

C2KV introduces a lightweight, trainable sidecar module called the C2 Extractor, which operates alongside a frozen base LLM. Instead of storing full, context-dependent KV caches, the C2 Extractor encodes documents into compact, position-agnostic representations called C2 KVs. This is achieved by injecting learnable C2 Tokens into the input stream, which act as memory slots for compressed information.

To ensure these compressed representations remain modular and effective, the framework employs a Structured Information Flow (SIF). This mechanism uses a block-structured attention mask to isolate original tokens from C2 Tokens during extraction, preventing semantic cross-talk and ensuring that the extracted KV states are position-independent. This allows the system to "plug and play" document segments at arbitrary positions within a prompt without requiring recomputation.

Co-Training and Performance

To bridge the gap between extraction and downstream reuse, C2KV utilizes a compression-concatenation co-training strategy. By training the model on generation tasks that require concatenating multiple compressed KV segments, the C2 Extractor learns to produce "merge-ready" states that maintain semantic coherence even when reordered. Experimental results demonstrate that this approach significantly reduces storage and transfer costs, achieving up to 17x inference speedup in long-context scenarios while preserving generation quality compared to traditional methods.