Token Radius Attention for Efficient Video Generation

Video Diffusion Transformers (VDiTs) rely on dense 3D self-attention to capture complex spatial and temporal dependencies, but the quadratic computational cost of this operation limits their scalability. Existing sparse attention methods often apply uniform computation budgets across entire heads or blocks, failing to account for the fact that different query tokens have varying levels of attention demand. This paper investigates whether token-level adaptive sparsity can be achieved efficiently without the overhead of explicit key ranking.

The authors introduce Token Radius Attention (TRA), a training-free framework that operates on two key insights. First, they observe that the number of keys required to capture a target attention mass (retained density) correlates log-linearly with attention entropy. Second, they find that dominant attention interactions form query-centered circular neighborhoods with radii that vary by token.

TRA implements this via an entropy-to-budget-to-radius pipeline:

1. Entropy-guided budgeting: It estimates a query-specific token budget using an analytic approximation of attention entropy.
2. Radius construction: It converts this budget into a query-specific base radius, which is then applied across frames using a temporal distance-decay rule to form regular spatiotemporal supports.
3. Efficient execution: To avoid the overhead of irregular masking, the authors introduce a tile-major token layout and a fused CUDA kernel that translates these token-level radii into hardware-compatible block-sparse masks for execution.

TRA demonstrates that token-level adaptivity can be achieved without sacrificing generation quality. Across seven configurations of Wan2.1, Wan2.2, and HunyuanVideo, the method retains only 9–19% of total attention interactions. This reduction in computation leads to a 1.56x–2.05x speedup in inference time while maintaining competitive video generation fidelity compared to dense baselines.