DistillAlign lets a Wan-1.3B video model beat a Wan-14B rival
Jiaxing Li and co-authors describe a problem with the standard way autoregressive video generation models are compressed into fast "distilled" students. The common pipeline uses Distribution Matching Distillation (DMD) across multiple stages, but treats the initialization stage and the DMD refinement stage as separate problems chasing different target distributions. Along the way, teams judge how good the intermediate student is mainly by visual quality scores such as VBench.
The authors argue this hides a real failure mode. Because the distribution matching loss used in DMD is mode-seeking by nature, a good initialization needs to match the mode coverage of the DMD teacher's distribution, not just look good on a visual score. To show this, they build a distributional evaluation protocol that measures precision and coverage between the student and teacher distributions in a shared latent space. Under this protocol, some initializations reach high precision but low coverage, which leads to worse results once refinement continues, while initializations that keep broader mode coverage refine better even though a visual score alone would not have flagged the difference.
The authors also find a second issue inside the DMD stage itself: even when the student's and teacher's target distributions are properly aligned at the start, DMD's reverse-KL training objective still pushes the student toward the teacher's highest-probability regions later in training, which reduces coverage and diversity over time.
To address both problems, the paper proposes "joint distillation," which combines DMD's mode-seeking objective with a Consistency-Distillation-based constraint that pulls training back toward mode coverage. In their experiments this combination improves generation quality, coverage and diversity together. The headline result: a system built around a comparatively small Wan-1.3B DMD teacher outperformed baselines that had been refined starting from the much larger Wan-14B, which the authors present as evidence that distributional alignment, not teacher size, is what drives the outcome.
Key facts
- Standard autoregressive video distillation pipelines run Distribution Matching Distillation (DMD) across separate initialization and refinement stages, judging the student mainly by visual scores like VBench.
- The paper introduces a distributional evaluation protocol measuring precision and coverage between student and teacher distributions in a shared latent space, showing some initializations score well visually but have low coverage.
- Even with aligned target distributions, DMD's reverse-KL objective still drives the student toward high-probability teacher regions late in training, cutting coverage and diversity.
- The proposed method, joint distillation, combines DMD's mode-seeking objective with a Consistency-Distillation-based mode-covering constraint.
- A system built with a Wan-1.3B DMD teacher outperformed baselines refined with the larger Wan-14B, which the authors attribute to distributional alignment rather than teacher size.
Why it matters
Compressing large autoregressive video generation models into fast distilled students is standard practice, and it normally relies on Distribution Matching Distillation (DMD) run across separate stages. Teams typically treat the initialization stage and the DMD refinement stage as pursuing different targets, and check the intermediate student mainly with visual quality scores such as VBench. The paper argues this hides a real failure: because DMD's loss is mode-seeking, a good initialization needs to match the DMD teacher's mode coverage, not just look good visually. Two students can score similarly on VBench while one keeps far less of the teacher's variety, and a visual-only check never catches it before refinement proceeds.
Who it affects
Teams building or compressing autoregressive video generation systems, particularly anyone distilling a large video diffusion-style teacher, here models in the Wan family, down into a smaller, faster student for production or research use.
How to use it
The paper proposes a distributional evaluation protocol that measures precision and coverage between student and teacher distributions in a shared latent space, meant to catch initializations that look fine on visual scores but have poor coverage before committing to full DMD refinement. It pairs this with joint distillation, a training method combining DMD's mode-seeking objective with a Consistency-Distillation-based mode-covering constraint, so the refinement stage does not lose diversity late in training.
How solid is it
The evidence is experimental. The authors report that with joint distillation, a system built on a comparatively small Wan-1.3B DMD teacher outperformed baselines refined starting from the much larger Wan-14B, presented as showing that distributional alignment, not teacher size, drives quality, coverage and diversity. The source does not state where this work has been reviewed or published, and it does not detail the benchmarks beyond quality, coverage and diversity.
Risks and caveats
The paper's own point is that visual scores like VBench can mask the exact coverage problem it identifies, so any single comparison, including the Wan-1.3B versus Wan-14B result reported in its own experiments, should be read with that same caution. The source gives no independent replication of that result outside the authors' own experiments.
“even with a Wan-1.3B DMD teacher, it outperforms baselines refined with Wan-14B, underscoring the importance of distributional alignment in autoregressive video distillation”
— the paper