Glonce

Intrinsic Distribution Matching adds a moment loss to DMD for few-step image models

researchThe Glonce editorsHugging Face🔥61Glonce rating61Impact50Novelty55Relevance100Surprise30Players35Credibility75
Intrinsic Distribution Matching adds a moment loss to DMD for few-step image models

The paper introduces Intrinsic Distribution Matching (IDM), a method for distilling text-to-image models into versions that generate in one step or a few steps. The problem it targets: with so few steps there is little room for refinement, so the model has to render the details in the prompt accurately the first time.

IDM builds on distribution matching distillation (DMD), which uses score differences to pull the generated distribution toward a reference distribution. Following prior work, the authors estimate that difference with internal velocity branches, which avoids training a separate fake-score network. They then ask whether an explicit moment constraint improves this learned score supervision. To answer it, they decompose the reverse-KL field into a Gaussian moment component and a non-Gaussian residual. That decomposition identifies statistics that can be constrained explicitly.

IDM keeps the full DMD correction and adds a loss that matches feature means and per-coordinate standard deviations. The features come from the generator's exponential-moving-average (EMA) copy, so no external feature encoder is needed.

The experiments use full-parameter tuning and low-rank adaptation (LoRA) tuning on two models: Z-Image-Turbo (6B parameters) and Qwen-Image (20B parameters). At two network evaluations, IDM reaches 94.25% on Chinese LongText-Bench and 90.20% on English LongText-Bench. The authors say these are the highest scores among the compared methods at this budget.

Controlled ablations show gains from the added moment loss. They also show that the generator's EMA features perform competitively with an external DINOv2 encoder across the five reported metrics. Code is available in the inclusionAI/IDM repository on GitHub.

Key facts

  • IDM keeps the full DMD correction and adds a loss matching feature means and per-coordinate standard deviations.
  • Features come from the generator's EMA copy, so no external feature encoder is required.
  • At two network evaluations it scores 94.25% on Chinese and 90.20% on English LongText-Bench, the highest among compared methods at that budget.
  • Experiments cover full-parameter and LoRA tuning on Z-Image-Turbo 6B and Qwen-Image 20B.
  • Ablations show gains from the moment loss; EMA features perform competitively with an external DINOv2 encoder across five reported metrics.

Why it matters

One- and few-step image generators have little chance to fix mistakes, so fine details such as rendered text are hard to get right. IDM tackles this by adding an explicit statistical constraint, matching feature means and standard deviations, on top of DMD's score-difference supervision. The reported LongText-Bench results suggest the added term helps text rendering at a budget of two network evaluations.

Who it affects

Researchers and engineers who distill diffusion or flow-based text-to-image models into fast few-step versions. The experiments use Z-Image-Turbo 6B and Qwen-Image 20B, so people working with models of those kinds are the closest audience. Because the method needs no external feature encoder, teams that want to avoid that extra component may find it relevant.

How to use it

Code is published at https://github.com/inclusionAI/IDM. The paper describes the setup as DMD with the full correction retained, plus a moment-matching loss computed on the generator's EMA copy. It reports using both full-parameter and LoRA tuning.

How solid is it

This is a preprint, and the claims come from the authors' own experiments. They report the highest LongText-Bench scores among compared methods at two network evaluations, and controlled ablations that show gains from the moment loss. The text does not give baseline scores, the size of the ablation gains, or which model and tuning mode produced the 94.25% and 90.20% figures. The five reported metrics are not named apart from LongText-Bench. Code release allows others to check the results.

Risks and caveats

The headline scores are tied to one budget of two network evaluations, and the comparison is limited to the methods the authors chose. The EMA features are described as competitive with a DINOv2 encoder, not better than it. No training cost, speed or latency figures are given, so the practical overhead of the added loss is unknown.