Yusheng Dai, Vladimir Iashin, Kangdi Wang, Yuxuan Jiang, Ilpo Viertola, Weiqiang Wang, Qiuhong Ke, Min Zhao, Jianfei Cai
8 min
Abstract
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/
Alex: They repeat the first frame wherever history is missing. That lets the encoder start without waiting for a full window of real frames. The encoder can still attend across the whole supplied segment, even though every real frame comes from the present or past. It needs no retraining for this conversion.
Sam: So they change which frames the encoder receives, rather than teaching it a new way to see. Where does the adjustable delay come from?
Alex: From how often they slide the window and extract new features. That interval is the hop size. A small hop gives frequent updates and a frontend look-ahead of 80 milliseconds. Larger hops reduce how frequently the visual encoder runs, saving computation without shortening its viewing window.
Sam: Could one trained sound generator handle those different update schedules? Changing the visual inputs might change more than just the computational bill.
Alex: The authors found that different hop sizes shift the visual feature distribution. Their Unified Varhop Framework adds a Synchronizer, a small network that aligns those features before sound generation. It maintains a constant output feature rate. A shared model then supports different hop sizes, with alignment close to separately trained specialists.
Sam: That addresses the visual side. But the opening claim also involved better sound quality. What changes in the audio generator?
Alex: The generator builds sound in short patches, using each completed patch to help predict the next. A local denoiser turns noise into each patch’s acoustic detail. In earlier streaming approaches, that denoiser refines the current patch alone. Earlier sound reaches it only indirectly through the generator’s internal state.
Sam: So the system remembers its past, but the detail-making component cannot directly inspect neighbouring sound. What do they give it instead?
Alex: Their default, History LocDiT, lets the local denoiser inspect previously generated clean patches. It gains direct acoustic context without waiting for anything new. In the component comparison, that improves one distribution-matching measure and semantic alignment, while leaving temporal alignment essentially unchanged. It does not improve every metric.
Sam: Looking backward costs no waiting. Looking forward presumably does, and the future sound has not been generated yet. How do they handle that dependency?
Alex: Their optional Future-Warmup LocDiT creates a provisional future patch. It uses the current internal state together with the next visual feature as an approximate condition. The current and provisional patches are refined together. Then the provisional patch is discarded and regenerated properly at the next step.
Sam: So that future patch helps shape the present, but it isn’t committed output. Is the added waiting included in the deployment claim?
Alex: The default system measured 156 milliseconds end to end on a single A800 graphics processor. The future-warmup option brings that to about 236 milliseconds, including its additional look-ahead. Both fall below the action-sound threshold reported in the authors’ study. That is a hardware-specific result, not a guarantee for every device.
Sam: Let’s separate fast delivery from convincing output. How broad was the training and evaluation, and what comparison best supports the alignment claim?
Alex: Training used about two thousand hours of video, combining in-the-wild and first-person footage. Main comparisons used the standard VGGSound test set. For controlled visual-encoder experiments, they also assembled roughly fifteen hundred held-out clips with strict audio-visual alignment. Other generation components were held fixed in that comparison.
Sam: That controlled comparison helps isolate the window idea. On the main benchmark, how close did the complete system get to offline timing?
Alex: LiveFoley’s higher-sample-rate version scored 0.48 on the predicted desynchronization measure, where lower is better. Offline MMAudio-L scored 0.47. The streaming baselines were worse. That supports near-offline alignment on this benchmark, rather than equal performance in every aspect of sound generation.
Sam: The timing measure itself comes from Synchformer, which also supplies LiveFoley’s visual features. Is there evidence beyond that model-based score?
Alex: There is a listening test. LiveFoley led the tested online systems in both perceived sound quality and audio-visual alignment, while approaching offline MMAudio-L in alignment. That supports the metric result. But the participant count and detailed listening protocol are not in this excerpt, so we cannot assess those here.
Sam: And the remaining quality gap matters if someone hears “near-offline” and assumes the whole problem is solved.
Alex: The authors explicitly retain that gap. LiveFoley trails offline MMAudio-L on several quality and distribution measures. Offline generation can refine all sound patches together; streaming must commit as observations arrive. Also, evaluation uses short clips. Although the method supports longer streaming, these results do not establish equal reliability across long interactive sessions.
Sam: Given those boundaries, who should read the full document, and where should they start?
Alex: Researchers building streaming generators should start with the Trailing Sliding Window method and the controlled visual-frontend comparison. Then read the denoising-window ablation to distinguish the default history method from the optional future method. Application developers should inspect Deployment and Extensions, especially the hardware-specific latency. For everyone else: low delay need not mean throwing away the past.
Sam: Keep the context; scrutinize the timing.