InstantFusion links denoising across frozen image generators
A paper listed on Hugging Face proposes InstantFusion, a lightweight shared latent interface that connects the denoising trajectories of heterogeneous image generators. The goal is to let denoising pass from one frozen generator to another without building a dedicated mapping for every pair of models.
The starting point is a gap the authors point to. Diffusion and flow models are now the de facto paradigms for image generation, and the resulting generators have increasingly heterogeneous architectures and representations. Yet the cross-model alignment of their latent representations, and the consistency of their underlying denoising dynamics, remain underexplored.
The authors report two observations. First, independently trained generators exhibit similar latent spatial organization. Second, directly handing off intermediate states between unaligned models can still preserve coarse visual content. InstantFusion is motivated by these two findings.
The method works like this. Model-specific Latent AutoEncoders (LAEs) project intermediate states from different generators into a shared latent space, and decode them into the latent space of a target generator. That shared space is constrained by two complementary objectives: aligning intermediate states derived from the same image at matched noise levels, and synchronizing their corresponding local denoising velocities.
Once trained, InstantFusion lets denoising transition across frozen generators without a dedicated mapping for every model pair. The authors demonstrate it in four settings: acceleration, multi-preference composition, cross-architecture on-policy distillation, and cross-model composition. They describe the performance across these settings as competitive.
Key facts
- InstantFusion is a lightweight shared latent interface that connects the denoising trajectories of heterogeneous diffusion and flow image generators.
- Model-specific Latent AutoEncoders (LAEs) project intermediate states into a shared latent space and decode them into the latent space of a target generator.
- The shared space is constrained by two objectives: aligning states from the same image at matched noise levels and synchronizing local denoising velocities.
- Once trained, denoising can move across frozen generators without a dedicated mapping for every model pair.
- The authors demonstrate it in four settings: acceleration, multi-preference composition, cross-architecture on-policy distillation and cross-model composition, with performance described only as competitive.
Why it matters
Image generation now rests on diffusion and flow models, and the field has produced many generators with different architectures and representations. The authors argue that how their latent representations line up, and how consistent their denoising dynamics are, has been little studied. InstantFusion is an attempt to exploit that: if independently trained generators already organize their latents in a similar spatial way, a shared interface may let them cooperate without a bespoke mapping for every pair.
Who it affects
The work is aimed at people who build or combine image generators. The four settings the authors demonstrate give the picture: acceleration, multi-preference composition, cross-architecture on-policy distillation and cross-model composition. Anyone working on those problems with more than one generator is the natural audience.
How to use it
As described, the recipe is to train a model-specific Latent AutoEncoder for each generator so that its intermediate states map into a shared latent space and can be decoded into a target generator's latent space. The shared space is trained with the two objectives of state alignment at matched noise levels and velocity synchronization. After that, denoising can hand off across frozen generators without a dedicated mapping for every model pair. No code, model or dataset release is mentioned.
How solid is it
This is a method paper known to us through its abstract, and the claims are the authors' own. Performance is only called competitive across the four settings; no benchmark scores, speedup factors or other quantitative results are given. The abstract does not name which generators or baselines were tested, and no comparison against specific prior methods is stated.
Risks and caveats
The authors' own motivating observation is limited: handing off intermediate states between unaligned models can still preserve coarse visual content, not fine detail. The result is described as competitive, not best. The abstract does not state training cost, parameter counts or the size of the LAEs beyond calling the interface lightweight, so the practical cost of adding a new generator to the shared space is unclear.