ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation

The quadratic growth of attention computation and key-value cache with respect to sequence length is a major bottleneck for ultra-long-context language models and high-resolution generative models. Existing efficient attention methods, linear approximations, or single-shot KV-cache compression techniques often face a structural contradiction: discarding fine-grained information degrades precise long-range retrieval, while keeping all details preserves high computational costs. This paper asks how a model can decouple information fidelity from global interaction cost without suffering irreversible information loss from one-shot compression.

The paper proposes ProxyFormer, a general architecture that organizes input features into two parallel streams: a fine-grained local stream that preserves local details across residual connections, and a compressed proxy stream where expensive global interactions occur at a fraction of the cost. In each layer, local features are compressed bottom-up into proxy states, global interactions are performed in the proxy space, and the updated proxies are decompressed and injected back into the local stream. Because the local stream persists across layers, details missed in one compression step remain accessible for later refinement, correcting compression bias through cross-layer iteration.

ProxyFormer relies on standard deep learning operators—such as reshapes, linear projections, and convolutions—avoiding the need for custom sparse kernels or indexers. To make large compression ratios stable and efficient, the authors introduce factorized multi-level compression and decompression, layer-wise dynamic compression ratios, asymmetric dual embeddings, and a proxy-only KV-cache inference scheme. During autoregressive decoding, historical local features can be offloaded once compressed into proxy states, meaning the model only needs to cache historical proxy states of length L/P alongside incremental generation tokens.