Researchers trace how audio and video 'leak' into each other in diffusion models
Audio-video diffusion models generate sound and picture together, coordinating them with text through cross-modal attention. A new study argues that this same coordination mechanism is also the source of a subtle failure: semantic leakage, where content meant to come from one modality bleeds into another and the output ends up not matching what was asked for.
The authors structure their analysis around what they call the 'attention triangle': the three cross-attention edges that connect the text, audio, and video streams inside the model. By probing each edge separately, they find that the audio-video edge behaves bidirectionally: audio can steer what the model generates for video, and video can just as easily steer what it generates for audio. This edge, they show, is shaped by biases baked into the model's own parameters, and it turns out to be a major source of leakage. When a prompt asks for something that conflicts with what the model has learned to expect, the cross-modal interaction along this edge can override the intended conditioning altogether, pulling the output toward a visually 'canonical' but incorrect result instead of the one the prompt actually specified.
The authors' broader point is that these artifacts are not simply attention wandering past its intended target. They are structured, bias-driven interactions traveling along specific, identifiable pathways inside the model. Working from that framing, the team extracts attention-derived signals that show how semantic content is distributed and grounded across the text, audio, and video streams. They use these signals two ways: as a diagnostic tool to both study leakage and deliberately trigger it under controlled conditions, isolating the contribution of individual cross-modal interactions, and as a guide for inference-time interventions, adjustments applied while the model is generating rather than requiring retraining, aimed at keeping the modalities more consistently aligned. The paper reports that extensive experiments back up the analysis and show improved semantic grounding while generation quality is preserved, though it does not give specific accuracy or leakage-rate figures for these improvements.
Key facts
- The study probes the 'attention triangle': the three cross-attention edges linking text, audio, and video inside audio-video diffusion models.
- The audio-video edge is bidirectional: audio can influence what gets generated for video, and video can influence what gets generated for audio.
- That edge is shaped by biases in the model's parameters and is identified as a major source of semantic leakage between modalities.
- When a prompt conflicts with the model's learned priors, cross-modal interaction along this edge can override the intended conditioning and push the output toward an incorrect but visually familiar result.
- The authors use attention-derived signals both to diagnose and deliberately induce leakage, and to drive inference-time interventions that improve cross-modal alignment without retraining, reportedly preserving generation quality.
Why it matters
Audio-video diffusion models are judged on whether sound and picture actually match what was asked for. This work identifies a concrete mechanism, biased routing along the audio-video attention edge, behind a failure mode where they quietly don't: the model substitutes a visually familiar but wrong outcome for the one the prompt requested. Naming the mechanism, rather than just observing the symptom, is what makes it something builders can target.
Who it affects
Anyone building or fine-tuning audio-video generation systems, and anyone using them for work where fidelity to the prompt matters rather than just plausibility of the output. The analysis is framed at the level of the underlying diffusion architecture rather than any single named product, so it applies broadly to systems built on cross-modal attention between text, audio, and video.
How to use it
The paper's proposed fix is an inference-time intervention guided by attention-derived signals, meaning it is applied while generating rather than requiring the model to be retrained. Teams with access to a diffusion model's cross-attention internals could use the same probing approach to check whether their own audio-video edge is biased before shipping.
How solid is it
The paper reports 'extensive experiments' supporting the analysis and showing improved semantic grounding with preserved generation quality, but the available text gives no accuracy figures, leakage-rate numbers, or named benchmarks to size those claims, and it does not specify which direction of the audio-video influence, audio-to-video or video-to-audio, is stronger.
Risks and caveats
The source text does not name the paper's authors, their institution, or the specific model or dataset used in the experiments, so the findings cannot yet be tied to a named system or verified against a public benchmark. Readers should treat the improvement as demonstrated in the authors' own setup rather than as a generally quantified fix until more detail is available.