LiveFoley generates audio for video in real time with a sliding-window encoder
A paper on Hugging Face presents LiveFoley, a streaming, causal, autoregressive diffusion architecture for video-to-audio (V2A) generation. The authors frame streaming V2A as essential for immersive interactive applications, and argue that real-time constraints leave existing streaming methods far behind offline models.
They name two bottlenecks. First, visual frontends shrink their receptive fields to cut look-ahead latency, which severely degrades audio-visual alignment. Second, causal autoregressive diffusion backends denoise each audio patch in isolation, which impairs audio quality. LiveFoley is built to address both with what the authors call imperceptible latency.
On the visual side, the paper introduces a Trailing Sliding Window. It converts non-causal, segment-level encoders into streaming modules without retraining. By realigning the window, it preserves precise audio-visual alignment, and a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on that tunable hop size, the authors propose the Unified Varhop Framework, meant to cover scenarios with different latency and computation budgets.
On the audio side, the authors explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost, and it is the default. Future-Warmup LocDiT uses an approximate future condition to break what the paper calls the autoregressive deadlock. That improves generation quality further, at the cost of an acceptable backend look-ahead.
The abstract concludes that extensive experiments show LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. A project page is linked at https://dalision.github.io/livefoley-page/.
Key facts
- LiveFoley is a streaming causal autoregressive diffusion architecture for real-time video-to-audio generation.
- A Trailing Sliding Window turns non-causal segment-level encoders into streaming modules without retraining, with a configurable hop size that keeps look-ahead below the human perceptual threshold.
- The Unified Varhop Framework, built on the tunable hop size, targets setups with different latency and computation budgets.
- On the audio backend, History LocDiT (the default) adds past context at no latency cost; Future-Warmup LocDiT uses an approximate future condition for further quality gains at the cost of some backend look-ahead.
- The authors say LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality.
Why it matters
Video-to-audio generation usually runs offline, with the whole clip available. The authors say real-time constraints leave streaming methods far behind those offline models, and they target that gap. Their pitch is that sound can be generated as video arrives, which they say is essential for immersive interactive applications. The paper names two causes of the gap: shrunken visual receptive fields that hurt alignment, and audio patches denoised in isolation that hurt quality.
Who it affects
The audience is researchers working on video-to-audio and streaming generative models. The abstract refers only to immersive interactive applications and names no specific products or uses. Builders of such applications are the implied beneficiaries if the claims hold up.
How to use it
The abstract points to a project page at https://dalision.github.io/livefoley-page/. It does not state a release of code, weights or a demo beyond that page. For anyone adapting the ideas, the paper describes a Trailing Sliding Window that wraps existing non-causal encoders without retraining, a hop size that can be tuned to a latency budget, and a choice between History LocDiT (default, no added latency) and Future-Warmup LocDiT (better quality, some backend look-ahead).
How solid is it
This is a research preprint, and the evidence here is the authors' own abstract. It says extensive experiments show LiveFoley matches offline models on temporal alignment and leads streaming systems on generation quality. No numbers, benchmarks, datasets or baselines are given in that text, so the size of the advantage cannot be judged from it. No concrete latency figure or hop size is stated either; the threshold is described only as the human perceptual threshold.
Risks and caveats
The claims of imperceptible latency and parity with offline models on temporal alignment come from the authors and are not backed by figures in the abstract. Model size, training data and hardware requirements are not stated. Future-Warmup LocDiT improves quality but adds backend look-ahead, so it trades away some of the latency advantage; the abstract does not say it is the better variant overall, and History LocDiT is the default.