NVIDIA and Bar-Ilan researchers close a cross-attention gap in video generation
A paper from Dvir Samuel (NVIDIA), Idan Schwartz (Bar-Ilan University) and Gal Chechik (Bar-Ilan University, NVIDIA) examines joint multimodal diffusion transformers, models that generate video together with a companion modality such as 3D motion or audio. The authors argue video carries richer information about physical events than these narrower companion signals, and they identify an asymmetry in how the model's cross-modal attention handles that gap: companion modalities develop strong correspondences to video, learning to follow what the video shows, but the reciprocal correspondences through which those modalities could constrain or correct the video remain substantially weaker. In other words, motion or audio ends up tracking the video without the video being properly held accountable to them. To fix this, the authors introduce RecCAR (Reciprocal Cross-modal Attention Regularization), a KL regularizer that aligns the weaker modality-to-video correspondence with the already well-established video-to-modality correspondence. Applied to joint video-motion generation, RecCAR raises the Human Anatomy score from 0.69 to 0.75. Applied to joint video-audio generation, it reduces the audio-video desynchronization score from 0.804 to 0.752. The authors report these gains come alongside overall improvements in generation quality while semantic alignment is preserved. Notably, the method does not require additional supervision or architectural changes to the underlying generator; it is applied as a lightweight LoRA parameter adaptation on top of existing pretrained models.
Key facts
- RecCAR (Reciprocal Cross-modal Attention Regularization) is a KL regularizer that fixes an asymmetry where companion modalities (motion, audio) attend strongly to video, but video does not attend back to them as strongly
- In joint video-motion generation, RecCAR raises the Human Anatomy score from 0.69 to 0.75
- In joint video-audio generation, RecCAR reduces the audio-video desynchronization score from 0.804 to 0.752
- The method needs no extra supervision or architectural changes, only a lightweight LoRA adaptation applied to existing pretrained generators
- Authors are Dvir Samuel (NVIDIA), Idan Schwartz (Bar-Ilan University) and Gal Chechik (Bar-Ilan University, NVIDIA)
Why it matters
Joint multimodal video generators are meant to produce video and a companion signal, like body motion or audio, that are mutually consistent. The paper shows this consistency is one-directional in practice: the companion modality learns to follow the video closely, but the video is not equally constrained by the companion modality, which can let it drift toward physically implausible results such as anatomically odd movement or audio that drifts out of sync. RecCAR targets that specific imbalance in cross-modal attention rather than the generator's overall quality.
Who it affects
The work is aimed at researchers and engineers building joint video-motion or video-audio diffusion transformers, and more broadly anyone working on multimodal generation systems where two or more signals need to stay physically consistent with each other.
How to use it
RecCAR is applied as a lightweight LoRA parameter adaptation on top of an existing pretrained joint generator, without changing its architecture or adding new supervision. The source does not mention a code or model release, so it is not clear whether an implementation is currently available.
How solid is it
The account comes from the paper's own abstract and reported results; the source text does not state a submission or publication date, nor does it name the specific datasets, benchmarks, or base model architectures behind the video-motion and video-audio evaluations beyond the task names themselves.
Risks and caveats
The reported gains, a Human Anatomy score moving from 0.69 to 0.75 and a desynchronization score dropping from 0.804 to 0.752, are measured on the two tasks the authors tested and may not generalize to other modality pairs or architectures. Without disclosed benchmark and dataset details, the improvements are hard to independently compare against other methods.
“companion modalities develop strong correspondences to video, but the reciprocal correspondences through which they constrain video remain substantially weaker”
— the paper's authors