Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/
Alex: Generating a sound quickly is not enough; it must land with the action. LiveFoley generates sound from live video with timing close to an offline model, while keeping measured delay low enough for interactive use. It still trails offline generation on several measures of sound quality.
Sam: So the claim is about making sound while the action unfolds, rather than adding it afterward. Whose work is this?
Alex: It’s a methods preprint by Yusheng Dai and colleagues, titled “LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds.” The intended uses include video games, augmented and virtual reality, and live camera feeds. The system takes incoming pictures rather than requiring a particular engine to produce them.
Sam: That makes the audience broader than researchers generating video itself. Let’s unpack what changes, whether the timing claim holds up, and what still separates this from offline sound generation.
Alex: Start with the timing budget. In the authors’ user study, delays in discrete action sounds, such as impacts, became perceptible at about 340 milliseconds. Speech and ambient sounds tolerated longer delays. That suggests different scenes need different balances between speed and computational cost.
Sam: But fast output can still be badly synchronized. What was stopping earlier systems from getting both speed and accurate timing?
Alex: The visual encoder, which turns pictures into useful features, faces a trade-off. Looking at a stretch of video helps it represent when something happens. But conventional streaming systems wait for that stretch to arrive before encoding it. That waiting is called look-ahead.
Sam: And shortening the stretch would remove some waiting, but also remove the motion context needed to place the sound.
Alex: That is what their controlled comparison shows. V-AURA uses a segment-level encoder called Synchformer, trained for audio-visual synchronization, but incurs substantial look-ahead. SoundReactor uses individual-frame features with little waiting, but weaker timing alignment. Shrinking Synchformer’s own viewing window also degraded alignment.
Sam: If the temporal context is necessary, how can LiveFoley keep it without waiting for future frames?
Alex: Move the window backward. Their Trailing Sliding Window ends at the current frame, so its entire contents are already available. As video arrives, the window slides forward. The encoder still processes a stretch of motion, but that stretch lies behind the present rather than ahead of it.
Sam: That sounds straightforward once the stream is running. At startup, though, there isn’t a history to look back over. What fills the window?
Alex: They repeat the first frame wherever history is missing. That lets the encoder start without waiting for a full window of real frames. The encoder can still attend across the whole supplied segment, even though every real frame comes from the present or past. It needs no retraining for this conversion.
Sam: So they change which frames the encoder receives, rather than teaching it a new way to see. Where does the adjustable delay come from?
Alex: From how often they slide the window and extract new features. That interval is the hop size. A small hop gives frequent updates and a frontend look-ahead of 80 milliseconds. Larger hops reduce how frequently the visual encoder runs, saving computation without shortening its viewing window.
Sam: Could one trained sound generator handle those different update schedules? Changing the visual inputs might change more than just the computational bill.
Alex: The authors found that different hop sizes shift the visual feature distribution. Their Unified Varhop Framework adds a Synchronizer, a small network that aligns those features before sound generation. It maintains a constant output feature rate. A shared model then supports different hop sizes, with alignment close to separately trained specialists.
Sam: That addresses the visual side. But the opening claim also involved better sound quality. What changes in the audio generator?
Alex: The generator builds sound in short patches, using each completed patch to help predict the next. A local denoiser turns noise into each patch’s acoustic detail. In earlier streaming approaches, that denoiser refines the current patch alone. Earlier sound reaches it only indirectly through the generator’s internal state.
Sam: So the system remembers its past, but the detail-making component cannot directly inspect neighbouring sound. What do they give it instead?
Alex: Their default, History LocDiT, lets the local denoiser inspect previously generated clean patches. It gains direct acoustic context without waiting for anything new. In the component comparison, that improves one distribution-matching measure and semantic alignment, while leaving temporal alignment essentially unchanged. It does not improve every metric.
Sam: Looking backward costs no waiting. Looking forward presumably does, and the future sound has not been generated yet. How do they handle that dependency?
Alex: Their optional Future-Warmup LocDiT creates a provisional future patch. It uses the current internal state together with the next visual feature as an approximate condition. The current and provisional patches are refined together. Then the provisional patch is discarded and regenerated properly at the next step.
Sam: So that future patch helps shape the present, but it isn’t committed output. Is the added waiting included in the deployment claim?
Alex: The default system measured 156 milliseconds end to end on a single A800 graphics processor. The future-warmup option brings that to about 236 milliseconds, including its additional look-ahead. Both fall below the action-sound threshold reported in the authors’ study. That is a hardware-specific result, not a guarantee for every device.
Sam: Let’s separate fast delivery from convincing output. How broad was the training and evaluation, and what comparison best supports the alignment claim?
Alex: Training used about two thousand hours of video, combining in-the-wild and first-person footage. Main comparisons used the standard VGGSound test set. For controlled visual-encoder experiments, they also assembled roughly fifteen hundred held-out clips with strict audio-visual alignment. Other generation components were held fixed in that comparison.
Sam: That controlled comparison helps isolate the window idea. On the main benchmark, how close did the complete system get to offline timing?
Alex: LiveFoley’s higher-sample-rate version scored 0.48 on the predicted desynchronization measure, where lower is better. Offline MMAudio-L scored 0.47. The streaming baselines were worse. That supports near-offline alignment on this benchmark, rather than equal performance in every aspect of sound generation.
Sam: The timing measure itself comes from Synchformer, which also supplies LiveFoley’s visual features. Is there evidence beyond that model-based score?
Alex: There is a listening test. LiveFoley led the tested online systems in both perceived sound quality and audio-visual alignment, while approaching offline MMAudio-L in alignment. That supports the metric result. But the participant count and detailed listening protocol are not in this excerpt, so we cannot assess those here.
Sam: And the remaining quality gap matters if someone hears “near-offline” and assumes the whole problem is solved.
Alex: The authors explicitly retain that gap. LiveFoley trails offline MMAudio-L on several quality and distribution measures. Offline generation can refine all sound patches together; streaming must commit as observations arrive. Also, evaluation uses short clips. Although the method supports longer streaming, these results do not establish equal reliability across long interactive sessions.
Sam: Given those boundaries, who should read the full document, and where should they start?
Alex: Researchers building streaming generators should start with the Trailing Sliding Window method and the controlled visual-frontend comparison. Then read the denoising-window ablation to distinguish the default history method from the optional future method. Application developers should inspect Deployment and Extensions, especially the hardware-specific latency. For everyone else: low delay need not mean throwing away the past.
Sam: Keep the context; scrutinize the timing.