LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

As large language models (LLMs) scale to million-token contexts, the quadratic computational cost of standard attention becomes a primary bottleneck. While DeepSeek Sparse Attention (DSA) mitigates this by selecting a subset of tokens to attend to, it suffers from two major system-level inefficiencies: high indexing overhead that scales poorly with context length and fragmented memory access patterns that prevent efficient hardware utilization.

To address these bottlenecks, the authors introduce LongCat Sparse Attention (LSA), which employs three co-designed strategies:

1. Streaming-Aware Indexing (SI): This strategy forces a portion of the attention budget into contiguous memory blocks (e.g., local sliding windows and fixed sinks). This improves memory coalescing and allows the hardware to utilize High Bandwidth Memory (HBM) more effectively.
2. Cross-Layer Indexing (CLI): Recognizing that important tokens are often consistent across adjacent layers, the authors group layers together. Only the first layer (the owner) performs the full indexing calculation, while subsequent layers (reuse layers) share these results, significantly reducing total compute.
3. Hierarchical Indexing (HI): This is a training-free, coarse-to-fine selection method. It first identifies relevant 'pages' of tokens using a coarse block-level score before performing fine-grained token selection only within those promising regions.

LSA allows for native training of models with context lengths up to one million tokens while maintaining performance parity with full-attention models. By optimizing the interaction between the indexing algorithm and the underlying hardware, LSA enables the development of larger models (such as the 1.6T-parameter LongCat-2.0) within limited compute budgets. The authors also open-sourced a 69B-parameter model, LongCat-Flash-Lite-Sparse, to facilitate further research into efficient long-context architectures.