Prompt Compression via Activation Aggregation

Modern LLMs often process the same system prompts or instructions repeatedly, leading to redundant computation. This paper investigates whether the task-relevant information contained in these prompts can be compressed into a single, fixed-size activation vector—a 'patch'—that can be injected into the model to replace the original token sequence without significant loss in performance.

The authors propose a two-pass framework. First, they extract hidden-state sequences from an intermediate layer of a frozen LLM. Second, they compress these sequences into a single patch vector using one of two methods: a Weighting MLP (W-MLP) that learns to assign importance weights to each token's activation, or a Transformer Compressor (TC) that uses an end-to-end encoder. Finally, this patch vector is injected into an early layer of the model by overwriting the hidden state of a neutral placeholder token (the UTF-8 replacement character).

The study reveals that a simple weighted sum of activations (W-MLP) is surprisingly effective, often outperforming the more complex Transformer Compressor. The W-MLP generalizes better to unseen prompts and maintains accuracy within 2% of the full-prompt baseline on instruction-following tasks. The authors also identify a consistent structural pattern in LLM activations: information is best captured when extracted from middle layers and injected into early layers, suggesting that early layers are highly receptive to integrated, high-level semantic representations.

This work provides a lightweight, computationally efficient alternative to KV-caching for fixed prompts. By demonstrating that task instructions can be distilled into a single latent vector, the authors offer a new perspective on how LLMs represent and process task-specific information, potentially enabling faster inference for systems that rely on long, static system prompts.