Krish Agarwal, Zhuoming Chen, Cheng Luo, Yongqi Chen, Haizhong Zheng, Xun Huang, Atri Rudra, Beidi Chen
9 min
Abstract
Real-time video generation with Diffusion Transformers is bottlenecked by the quadratic cost of 3D self-attention, especially in real-time regimes that are both few-step and autoregressive, where errors compound across time and each denoising step must carry substantially more information. In this setting, we find that prior sparse-attention approximations break down, despite showing strong results for bidirectional, many-step diffusion. Specifically, we observe that video attention is not reliably sparse, but instead combines pronounced periodic structure driven by spatiotemporal position with dynamic, sparse semantic correspondences and dense mixing, exceeding the representational capacity of even oracle top-k attention. Building on this insight, we propose Monarch-RT, a structured attention parameterization for video diffusion models that factorizes attention using Monarch matrices. Through appropriately aligned block structure and our extended tiled Monarch parameterization, we achieve high expressivity while preserving computational efficiency. We further overcome the overhead of parameterization through finetuning, with custom Triton kernels. We first validate the high efficacy of Monarch-RT over existing sparse baselines designed only for bidirectional models. We further observe that Monarch-RT attains up to 95% attention sparsity with no loss in quality when applied to the state-of-the-art model Self-Forcing, making Monarch-RT a pioneering work on highly-capable sparse attention parameterization for real-time video generation. Our optimized implementation outperforms FlashAttention-2, FlashAttention-3, and FlashAttention-4 kernels on Nvidia RTX 5090, H100, and B200 GPUs respectively, providing kernel speedups in the range of 1.4-11.8X. This enables us, for the first time, to achieve true real-time video generation with Self-Forcing at 16 FPS on a single RTX 5090.
Alex: Right, so the permutation aligns everything to expose low-rank blocks that match how video patches naturally connect. And tiling—what's that add to make it more precise?
Sam: Without tiling, the blocks might be too big to catch fine details, like subtle links between distant but similar objects. Tiling breaks those blocks into smaller sub-blocks, each handled separately, so the model can represent sharper semantic ties—like a person linking to a specific car across frames—while keeping the overall repeating structure. It's like zooming in on sections of a photo for better compression without losing the big picture. The paper shows it approximates full attention closely, even at high sparsity.
Alex: So by aligning and subdividing, it handles both the predictable repeats and the irregular important connections.
Sam: Originally, MonarchAttention refines by alternating updates between L and R factors over several steps, tweaking one while fixing the other to match the ideal attention without building the full dense map. But 10 steps slow things down for real-time use. They finetune the model so one step matches that quality, as visuals in the paper confirm. Combined with custom kernels, this drops computation sharply while holding quality.
Alex: Oh—so finetuning cuts the iterations without retraining from scratch. And the upshot for real-time generation?
Sam: It enables 16 frames per second on one RTX 5090 card with Self-Forcing, about 12 times faster than some prior attention kernels, without quality drop even at 95% less computation. The evidence points to MonarchRT preserving fidelity where sparse methods distort geometry from error buildup.
Alex: So it holds quality at high speed because it matches the true structure of video attention. But how does this factorization handle those tricky semantic links without messing up the positional repeats?
Sam: Video attention splits into two main parts: smooth drops in connection strength as things get farther apart in space or time—like how nearby pixels in a frame or close frames matter more—and rare, strong jumps to distant but meaningful spots, like a moving car linking across seconds. After rearranging with permutation, each block in the map is simple enough to capture with one basic pattern, either pure distance decay, one jump, or jumps all from or to the same spot—what's called an attention sink, where everything funnels to a key token.
Alex: Okay, so blocks stay simple post-rearrange—one pattern per block for positions or single/focused semantics. That relies on the blocks lining up right with the video's layout, right?
Sam: Yes—the paper shows misalignment mixes unrelated tokens into blocks, ruining that simplicity and distorting output, like grouping puzzle pieces from different areas. They fix it by picking block sizes that match video dimensions exactly, say one for frames-times-height and the other for width, so each block follows natural lines across time and space. Tiling then splits blocks smaller for finer jumps.
Alex: So alignment prevents the mixing that sparse methods can't avoid.
Sam: Precisely. Misalignment leads to full-rank blocks needing dense compute; alignment keeps them low-rank. The paper contrasts examples: aligned blocks recover clean patterns, misaligned warp them. This structured fit, plus tiling and one-step finetuning, enables the 95% sparsity at no quality loss.
Alex: So alignment is crucial because splitting a video dimension across blocks mixes things up, preventing the clean factorization. Walk me through why that matters mechanically—like, what goes wrong inside the math?
Sam: Picture the video's layout as three lengths: frames over time, height, and width. If you try to split any one length across two block sizes, like chopping a row halfway, the connections inside can't separate neatly into the L and R parts anymore. The paper stresses no dimension can straddle both block sizes—each must go fully to one.
Alex: Okay, so full dimensions per block keep the separation possible, avoiding that mixing. And that's why misaligned choices distort visuals?
Sam: Exactly—as their figure shows, aligned blocks like frames-height by width hold sharp images, but misaligned ones warp geometry badly. Tiling builds on it by splitting those blocks into smaller tiles, each with its own mini-factorization. The paper proves tiled Monarch can represent everything plain Monarch can, plus more—strictly higher expressivity for the same big block sizes.
Alex: So tiling adds precision for semantics without bloating costs, and alignment ensures it works. The evidence from their scores shows it holds up against full attention?
Sam: The tables confirm MonarchRT at 95% sparse matches dense attention's quality and semantics scores closely on Self-Forcing tasks—a clear preservation. The evaluations on VBench and other metrics show MonarchRT outperforming baselines like SVG and RadialAttention, staying neck-and-neck with dense attention overall.
Alex: Right, and the speedups translate to real-world use—like 16 frames per second on a single consumer GPU. But are there trade-offs, like needing special training or hardware tweaks?
Sam: A key one is that optimal sparsity requires finetuning the model, rather than dropping it in training-free—though it still beats baselines there too. Block alignment ties to specific resolutions and hardware, so adapting to new setups needs recalibration.
Alex: Okay, so it's not plug-and-play everywhere—finetuning and alignment choices are musts for peak performance. Still, that enables real-time on everyday high-end cards.
Sam: Exactly. The paper positions this as a step toward interactive editing, where you tweak videos live on consumer GPUs, or AR and VR content that generates on the fly with quality matching slower diffusion models. It addresses the core mismatch in video attention, paving a practical path forward without the distortions from cruder approximations.
Alex: Well put. By aligning with video's true geometry through factorization, tiling, and optimized kernels, MonarchRT delivers meaningful efficiency for real-time generation. The evidence suggests it's a clear advance. Thanks for listening to ResearchPod.