Video DeltaNet speeds MiniMax H3 video generation 14.5x
Video diffusion models denoise long sequences of spatiotemporal tokens, and the researchers behind this paper say that repeated attention over those sequences is a major computational bottleneck. Linear attention, already widely used in large language models, looks like an obvious fix, but the authors find that applying it directly to video models fails to preserve the fine-grained interactions that high-quality video generation needs.
Their answer is Video DeltaNet (VDN), which combines local Softmax attention with a bidirectional linear-memory branch for long-range context across the video. The linear branch, called Video Delta Attention (VDA), updates its memory once per frame, jointly incorporating that frame's spatial tokens rather than treating tokens independently. Separate output projections and learnable gates calibrate how the two branches combine, and a staged teacher-alignment recipe introduces the new linear pathway into an already pretrained model gradually rather than training it from scratch.
The authors instantiated VDN on MiniMax H3, applying the hybrid attention only to video-to-video interactions while keeping the original Softmax attention for interactions involving text or audio. Paired with eight-step distillation and an optimized SGLang serving stack, the resulting VDN-H3 model completes the diffusion transformer's denoising for a 14.3-second, 768p video in 6.70 seconds on eight NVIDIA B200 GPUs. The paper reports this as a 14.5x speedup over the 50-step dense H3 baseline running on the same eight-GPU setup.
Key facts
- Video DeltaNet (VDN) combines local Softmax attention with a bidirectional linear-memory branch (VDA) for long-range video context.
- VDA updates its memory once per frame, jointly incorporating that frame's spatial tokens.
- Instantiated on MiniMax H3, the hybrid attention is used only for video-to-video interactions; Softmax attention is retained for text or audio interactions.
- With eight-step distillation and an optimized SGLang serving stack, VDN-H3 denoises a 14.3-second, 768p video in 6.70 seconds on eight NVIDIA B200 GPUs.
- That is a 14.5x speedup over the 50-step dense H3 baseline on the same GPU count.
Why it matters
Attention cost scales with the length of the token sequence, and video diffusion models process much longer spatiotemporal sequences than text models do, which the authors identify as the main bottleneck to fast denoising. Linear attention cuts that cost in language models but loses the fine-grained detail video generation depends on when applied naively. VDN's hybrid design, mixing local Softmax attention with a linear-memory branch built specifically for video, aims to keep that detail while still cutting compute, which matters for any video model that needs to generate long clips, such as livestream video generation, without paying full quadratic attention cost throughout.
Who it affects
The direct beneficiary demonstrated in the paper is MiniMax H3, whose denoising the authors sped up with VDN. More broadly, the method targets teams building or serving video diffusion transformers that need long-range temporal context, since the staged teacher-alignment recipe is designed to retrofit the hybrid attention onto an existing pretrained model rather than requiring training from scratch.
How to use it
VDN is applied through a staged teacher-alignment recipe that introduces the linear-memory pathway into a pretrained video model progressively, then benefits further from distillation, in this case eight steps, and an optimized serving stack, here SGLang, to reach the reported inference speed. The source gives no release date, no code availability and no statement on whether model weights will be published.
How solid is it
The headline number, a 14.5x speedup, comes from one comparison: VDN-H3 against a 50-step dense H3 baseline on the same eight-GPU hardware, generating the same 14.3-second, 768p video. The source does not include a comparison against any other linear-attention video model, nor does it give training data, dataset details, or human or automated visual-quality evaluation results alongside the speed numbers. The paper had drawn 23 points and 2 comments on Hugging Face's papers board at the time it was captured.
Risks and caveats
The source text gives no author names, institutional affiliations or publication venue beyond the paper itself, no training or evaluation data, no comparison to other linear-attention approaches for video, and no release date or statement on code or weights availability. Speed gains from swapping attention mechanisms have shipped before without matching quality claims holding up under independent testing, and nothing here confirms visual quality was preserved beyond the authors' own description of their method.