Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, Eli Shechtman
8 min
Autoregressive (AR) video diffusion models often struggle with exposure bias—a mismatch between the clean, ground-truth data seen during training and the imperfect, model-generated outputs encountered during inference. This cumulative error leads to degraded visual quality as generation progresses. The authors investigate whether aligning the training process with the autoregressive inference process can resolve this issue and enable efficient, real-time video generation.
To bridge the train-test gap, the authors introduce Self Forcing, an algorithm that trains models by unrolling the autoregressive process. Unlike Teacher Forcing (which conditions on ground-truth frames) or Diffusion Forcing (which conditions on noisy context), Self Forcing generates each frame conditioned on the model's own previous self-generated outputs.
Because naive autoregressive unrolling is computationally expensive, the approach employs two key strategies:
The framework also includes a rolling key-value (KV) cache mechanism, enabling efficient long-video extrapolation without the need to recompute attention matrices at every window shift.
Self Forcing consistently outperforms existing autoregressive and diffusion-based baselines in both objective VBench scores and human preference studies. By optimizing the entire sequence rather than independent frames, the model better handles temporal dependencies and error accumulation. Furthermore, the approach proves unexpectedly efficient; because it avoids the specialized (and often slower) attention masking required by parallel training paradigms, it achieves real-time throughput (17 FPS) with sub-second latency on a single H100 GPU.
This work suggests that the current reliance on fully parallelizable training for transformers—while beneficial for scaling—may be fundamentally misaligned with the causal requirements of sequential media generation. By demonstrating that an autoregressive post-training phase can drastically improve quality and latency, this study provides a blueprint for building high-fidelity interactive video simulators and live-streaming tools.
We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models. It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs during inference. Unlike prior methods that denoise future frames based on ground-truth context frames, Self Forcing conditions each frame's generation on previously self-generated outputs by performing autoregressive rollout with key-value (KV) caching during training. This strategy enables supervision through a holistic loss at the video level that directly evaluates the quality of the entire generated sequence, rather than relying solely on traditional frame-wise objectives. To ensure training efficiency, we employ a few-step diffusion model along with a stochastic gradient truncation strategy, effectively balancing computational cost and performance. We further introduce a rolling KV cache mechanism that enables efficient autoregressive video extrapolation. Extensive experiments demonstrate that our approach achieves real-time streaming video generation with sub-second latency on a single GPU, while matching or even surpassing the generation quality of significantly slower and non-causal diffusion models. Project website: http://self-forcing.github.io/
Alex: Evaluating whole videos sounds useful, but previous work also did that. Where is the actual difference from CausVid, the close predecessor discussed here?
Sam: CausVid already combined fast sequential generation with distribution matching, which trains generated outputs to resemble a target video distribution. But it produced its training outputs using Diffusion Forcing. The authors' criticism is that those outputs don't follow the same distribution as its actual sequential generation.
Alex: So the novelty isn't simply judging a whole clip. It's judging clips produced through the same self-dependent process the model will use later.
Sam: That's the key distinction. One concrete failure in the comparisons is progressive over-saturation in CausVid's videos. Self Forcing trains while exposed to its own generated history, and shows less of that deterioration. The controlled experiments test whether that change matters beyond individual examples.
Alex: Generating your own training inputs sounds expensive. How do they avoid making every update a long video-generation job with an enormous memory bill?
Sam: They use four denoising steps, rather than a long diffusion chain. They also truncate gradients, meaning they limit which computations receive the training signal. For each frame, learning passes through only the last denoising step executed during that training rollout.
Alex: If only the last step learns, wouldn't the earlier denoising steps miss out?
Sam: The stopping point varies randomly between training iterations, so different steps receive supervision. They also stop gradients from flowing backward through earlier frames. That saves memory, but it means this isn't unrestricted learning through the entire generated history.
Alex: They also reuse earlier computations. What exactly gets stored, and why does that matter during training?
Sam: A key-value cache stores attention information from earlier frames so the model can reuse it. Self Forcing uses that cache during training as well as generation. And although frames arrive sequentially, the model still processes the contents within each frame or group in parallel.
Alex: Let's move from the mechanism to the evidence. Which result best separates a genuine training improvement from simply using a faster generator?
Sam: The strongest comparison keeps speed fixed. Self Forcing and the official CausVid implementation both deliver seventeen frames per second on a single H100 graphics processor. They also share a first-frame latency of about seven-tenths of a second, but Self Forcing scores better on quality.
Alex: How large is that quality difference, and what does the score measure?
Sam: VBench is a video-generation benchmark assessing visual quality and alignment with the requested content. Self Forcing's overall score is about eighty-four, compared with about eighty-one for CausVid. I wouldn't translate that directly into a percentage improvement in perceived quality.
Alex: The starting model also matters. If the fast version loses what made the original good, beating another fast method wouldn't settle the question.
Sam: Their starting point is Wan's pretrained video diffusion model. Self Forcing essentially matches its overall benchmark score, while delivering video much sooner. Human preference comparisons also favor Self Forcing over the tested alternatives, including that starting model.
Alex: How much weight should we put on those preferences? And do they isolate the training change?
Sam: They're supporting evidence, but the supplied document doesn't give a participant count. The controlled ablations are more informative about the mechanism. Under the authors' training setup, Self Forcing beats Teacher Forcing and Diffusion Forcing, including versions using the same distribution-matching objective.
Alex: They also test generating individual frames versus groups of frames. That seems useful because individual-frame generation repeatedly feeds predictions back into the model.
Sam: The conventional training approaches degrade more in that setting. Self Forcing maintains similar quality across both variants, supporting the error-accumulation argument. It also works with several whole-video training objectives, suggesting the benefit isn't confined to one particular loss.
Alex: What happens when the video outlasts the history the model can keep? Streaming needs more than a good short clip.
Sam: They introduce a rolling cache: retain recent frames' stored information and evict the oldest entries. That avoids recomputing the overlapping history. But a naive version flickers, because the first encoded frame has different properties from later frames.
Alex: How do they stop the model depending on that special first frame forever?
Sam: During training, they prevent the final group of frames from attending to the initial group. That rehearses losing access to the beginning. It reduces the artifacts while preserving throughput, but it doesn't eliminate the broader long-video limitation.
Alex: So what's the boundary on the claim, and who should spend time with the full paper?
Sam: Training clips are five seconds long, and the authors still observe degradation substantially beyond the training context. Gradient truncation may also limit long-range learning. I'd read this for sequential generative modeling, starting with the self-rollout method and controlled ablations; for streaming, add the rolling-cache section and limitations. Everyone else can keep one principle: train a generator on the imperfect past it will actually have.
Alex: That's a useful distinction to carry into the next model you evaluate.