Diffusion language models, explained from first principles

The Kuleshov Group blog has published a long-form tutorial on diffusion language models: how they work, and the research advances that produced today's diffusion LLMs. The post is adapted from workshop talks and lectures given at ICLR 2026 and MLSS 2026, and no individual author is credited in the visible text.

The piece opens by framing two competing families of generative AI. For continuous data such as images or video, the standard approach is diffusion. For discrete data such as text or code, the standard approach has instead been autoregressive generation: producing tokens one at a time, left to right, each conditioned on everything before it. That approach works, but has structural limits. Diffusion offers a different recipe: generate the whole sequence at once from an initial guess, then iteratively refine it over a number of steps. This lets a system trade speed for quality by using fewer or more steps, lets mistakes made early on be corrected later, and lets every step attend to context on both sides of a token rather than only the tokens already generated. The tutorial says applying diffusion to language was an open problem for a long time, that 2024 was a turning point when diffusion models became competitive with autoregressive models on quality, and that by 2026 diffusion LLMs are shipping from industry labs, citing Mercury 2 from Inception Labs as an example.

To build the idea from the ground up, the tutorial first reviews Gaussian diffusion for images. A forward process takes a clean image and repeatedly mixes in random Gaussian noise until only static remains; this step needs no learning, but it manufactures unlimited training data. A reverse process is then trained to invert that trajectory: given a noisy image, predict the noise (or, equivalently, the clean image), strip it away, and repeat until a clean sample emerges.

The obstacle in porting this to language is that continuous Gaussian noise is not well defined for discrete tokens, so the tutorial substitutes masking as the notion of noise. This gives masked diffusion, which it says the authors' own group helped popularize. A masked diffusion language model (MDLM) is trained by taking clean sequences, masking a random fraction of tokens, and training a bidirectional transformer to fill in the blanks: essentially BERT, but with a randomized masking rate at training time and, unlike BERT, generative at inference time. Generation starts from a fully masked sequence and repeatedly asks the model to fill in and lock in some of the remaining blanks, converging on a full sample after many rounds.

The tutorial then works through the math. The forward process is governed by a schedule alpha_t, the probability a token remains unmasked, running from 1 (clean) to 0 (fully masked); it is implemented as a Markov chain over increasingly masked versions of the sequence. Using Bayes' rule, the ideal reverse process can be derived when the clean sequence is known; in practice a model x_theta is trained to predict the clean sequence from a partially masked one and stands in for the unknown true sequence in that formula. Assessing the model's quality means approximating the (intractable) log-likelihood with an evidence lower bound (ELBO), which for masked diffusion works out, in the simplified case where alpha_t = 1-t, to the same cross-entropy loss BERT uses, but averaged over every masking rate rather than one fixed rate, and normalized by the expected fraction of masked tokens. That ELBO also gives a principled way to compare diffusion and autoregressive models by perplexity; the tutorial notes there was long a substantial perplexity gap between the two families, and that simplified masked diffusion models were among the first to close much of it.

The article is explicit that plain MDLMs as defined are useful for building intuition but are not production-ready: they generate only fixed-length sequences, do not support iterative refinement (error correction) out of the box, and are not especially fast without further post-training. It then covers extensions used in real open-weights diffusion LLMs. Block diffusion fixes the fixed-length problem by running diffusion over blocks conditioned on previously generated tokens, letting the block size be tuned to the task, for instance to a known interaction length in biological data, or to maximize GPU utilization by matching the arithmetic intensity of the hardware. Block diffusion also supports KV caching, the same technique that speeds up autoregressive generation: once a block is generated, its keys and values can be cached and reused for later blocks. Other approaches to variable-length generation build on connections between masked diffusion and any-order autoregressive models; the tutorial names Set Diffusion, which extends block diffusion to arbitrary sets of positions rather than strict left-to-right blocks, and mentions Edit Flows as a further approach. Finally, it notes that standard masked diffusion models are encoder-only, like BERT, in contrast to the decoder-only architecture of autoregressive models like GPT, and that encoder-only sampling requires invoking the full network at every denoising step, which can be computationally expensive. The visible text cuts off mid-sentence immediately after introducing 'a key insight' meant to address that cost, so the tutorial's proposed fix is not captured here.

Key facts

Diffusion language models generate an entire sequence at once and iteratively refine it, instead of producing tokens one at a time left to right like autoregressive models.
The tutorial dates 2024 as the turning point when diffusion models became competitive with autoregressive models on quality, and says diffusion LLMs are shipping from industry labs by 2026, citing Mercury 2 from Inception Labs.
Masked diffusion language models (MDLM) train a bidirectional transformer to fill in randomly masked tokens, like BERT with a randomized masking rate, but generate text by starting from a fully masked sequence and unmasking it step by step.
The evidence lower bound (ELBO) for masked diffusion reduces to the same cross-entropy loss BERT uses, averaged over all masking rates and normalized by the expected fraction of masked tokens, and it lets diffusion and autoregressive models be compared directly by perplexity.
Block diffusion extends masked diffusion to variable-length generation by diffusing over blocks conditioned on prior tokens, and it supports KV caching, the caching technique that speeds up autoregressive generation.

Why it matters

Mainstream language models are autoregressive, generating one token at a time conditioned on everything before it. Diffusion offers a structurally different alternative: generate the whole sequence at once from an initial guess and iteratively refine it, which lets a system trade speed for quality with the number of steps, correct earlier mistakes, and let every step see context on both sides of a token. The tutorial frames 2024 as the point where diffusion language models became competitive with autoregressive ones on quality, and 2026 as the point where they are shipping from industry labs, naming Mercury 2 from Inception Labs. That makes diffusion a live second paradigm for text generation rather than a purely academic curiosity.

Who it affects

The audience is people building or studying language models: ML researchers and engineers who want to understand or implement a diffusion LLM, and anyone evaluating diffusion as an alternative generation strategy to autoregressive decoding. It is not a product announcement and names no company or user base directly affected; Mercury 2 and Inception Labs appear only as an example of an existing diffusion LLM release.

How to use it

The material is a self-contained tutorial, adapted from workshop talks and lectures at ICLR 2026 and MLSS 2026, meant to be read start to finish: it builds from Gaussian diffusion for images, to masked diffusion for text (MDLM) with its forward/reverse process and ELBO derivation, to production extensions like block diffusion and KV caching. It targets readers who want to actually build a diffusion language model rather than only consume one, walking through the schedule alpha_t, the Markov chain formulation, and the loss function in enough detail to implement from.

How solid is it

The post runs on the Kuleshov Group's own blog and is explicitly adapted from conference and workshop lectures, so its math (the forward/reverse process derivation, the ELBO, its equivalence to a generalized BERT loss) is standard, previously presented material rather than new experimental results. No individual author is named in the visible text, and no benchmark numbers, accuracy figures, or perplexity values are given anywhere in what was retrieved, so claims like closing 'much of' the perplexity gap cannot be checked against a figure here.

Risks and caveats

The visible text cuts off mid-sentence right after introducing what it calls 'a key insight' meant to address the computational cost of encoder-only sampling in masked diffusion, so that part of the argument is missing from this account. The tutorial gives no numbers to quantify how much of the diffusion-versus-autoregressive perplexity gap has closed, and it does not name the identity of 'our group' credited with popularizing masking-based diffusion or any individual author of the post.

“Applying diffusion to language had long been an open problem. In 2024 the field reached a turning point, as diffusion models became competitive with autoregressive models on quality.”

— "How to build a diffusion language model," Kuleshov Group blog