GenFirst trains latent generative models without collapse
Latent generative models such as latent diffusion normally follow a two-stage pipeline: first a variational autoencoder (VAE) is trained to reconstruct images, then a generative model is trained on top of that VAE's frozen latent space. The researchers point out that this is a compromise, because latents optimized for reconstruction are not necessarily good for generation. Training both models jointly, end to end, is appealing but has been hard to make work: direct joint training tends to suffer from latent collapse and from a conflict between the reconstruction and generation objectives.
The authors trace the problem to two mechanisms. First, the entropy term inside the Kullback-Leibler divergence objective turns out to be essential for preventing collapse: reconstruction and prior fitting both push the posterior distribution to shrink, while the entropy term is what keeps the latent space from becoming degenerate. Second, reconstruction and generation learn at different speeds: reconstruction is fast and strongly supervised, while generation is slower and harder to optimize.
Based on these two insights, the authors propose GenFirst, a generation-before-reconstruction training strategy. The generative objective is given priority first, shaping the latent space while reconstruction pressure is kept weak; reconstruction is then strengthened progressively afterward to recover fine visual detail. According to the authors, this is the first time direct end-to-end training of both models has been achieved without latent collapse.
GenFirst was tested with two different families of generative priors: continuous autoregressive priors, which have exact likelihoods, and SiT priors, which have implicit likelihoods. With the end-to-end objective and GenFirst, a SiT-based model reaches a gFID of 0.97 with classifier-free guidance (CFG) and 1.45 without CFG on ImageNet-256, and an MMDiT-based model reaches a GenEval score of 0.90 on text-to-image generation. The authors also report extending the same framework beyond image generation, to shared visual latents used for both generation and representation learning, and to continuous unified text-image generation, though the text gives only a one-sentence description of those extensions.
Key facts
- Standard latent generative pipelines train a VAE for reconstruction, then freeze its latent space and train a generator on top; the authors argue reconstruction-optimized latents are not necessarily generation-friendly.
- Two insights drive the fix: the entropy term in the KL divergence objective prevents latent collapse, and reconstruction and generation learn at different speeds (reconstruction fast, generation slower).
- GenFirst trains the generative objective first under weak reconstruction pressure, then strengthens reconstruction progressively, achieving what the authors call the first direct end-to-end training without latent collapse.
- On ImageNet-256, a SiT-based model reaches a gFID of 0.97 with CFG and 1.45 without CFG; an MMDiT-based model scores 0.90 on GenEval for text-to-image generation.
- The authors say the same framework extends to shared visual latents for representation learning and to continuous unified text-image generation, described in one sentence in the text.
Why it matters
Latent diffusion and similar models usually rely on a frozen, separately trained VAE, a compromise the field has largely accepted because joint training kept collapsing. This paper reports the first direct end-to-end training of both the reconstruction and generation objectives without that collapse, which reopens the question of whether the frozen two-stage pipeline was ever necessary.
Who it affects
Researchers and engineers building latent diffusion or other latent generative models, including autoregressive image generators and text-to-image systems, plus anyone working on continuous unified text-image generation or on shared visual latents for representation learning.
How to use it
GenFirst is presented as a training strategy, not a released product; the text does not state a code or model release. Applying it means running the generative objective first under weak reconstruction pressure so it shapes the latent space, then progressively strengthening reconstruction, while keeping the entropy term in the KL divergence objective so the posterior does not collapse.
How solid is it
The method was validated on two distinct prior families, continuous autoregressive priors with exact likelihoods and SiT priors with implicit likelihoods, and reports concrete numbers: a gFID of 0.97 with CFG and 1.45 without CFG on ImageNet-256, and a GenEval score of 0.90 for text-to-image generation. The text gives no side-by-side numbers for standard two-stage baselines, so the size of the improvement over the conventional pipeline cannot be judged from this material alone.
Risks and caveats
The text names no authors, institutions, publication venue, release date, training cost, or hardware. The extensions to shared visual latents and to continuous unified text-image generation get only a one-sentence mention with no supporting detail. The claims about the entropy term and about asymmetric learning dynamics are the authors' own account of their analysis, not an independently verified result.
“we achieve the first direct end-to-end training without latent collapse”
— GenFirst paper