Glonce

Sander Dieleman traces the comeback of continuous diffusion language models

researchThe Glonce editorsHacker News🔥66Glonce rating66Impact55Novelty50Relevance95Surprise60Players70Credibility80
Sander Dieleman traces the comeback of continuous diffusion language models

Modern large language models generate text one token at a time, an autoregressive process that trains efficiently in parallel but is not the only way to build a generative model for sequences. Diffusion offers another route: instead of predicting the next token, a diffusion model learns to reverse a corruption process that gradually destroys a signal, typically by adding Gaussian noise until it swamps the original data. In a retrospective posted to his blog, researcher Sander Dieleman traces how this idea was applied to language and argues that one branch of it, continuous diffusion, is showing signs of a comeback after several quiet years. The first attempts to bring diffusion to language arrived in 2021, before ChatGPT existed, using discrete corruption processes suited to categorical token data: multinomial diffusion, D3PM and SUNDAE. Discrete diffusion promised to fix theoretical weaknesses of autoregression, such as exposure bias from teacher forcing and the difficulty of infilling and constrained generation. In 2022, a separate line of work took a different tack: represent discrete tokens as continuous embedding vectors, then apply the same Gaussian diffusion machinery that was already working well for images. Diffusion-LM opened this continuous branch, and a wave of follow-up papers arrived within months, including two Dieleman co-authored himself, self-conditioned embedding diffusion (SED) and continuous diffusion for categorical data (CDCD). Then, by Dieleman's account, continuous diffusion for language effectively went extinct. A diagram from a 2025 survey paper, which he shared on social media and describes as leaving him "a bit wistful," shows that virtually all new diffusion-language-model research after 2023 used discrete methods instead. He offers this as speculation rather than settled fact: one possible driver was the "ChatGPT moment," which he says shifted the field's priorities from theoretical elegance toward matching autoregressive models on raw performance, where discrete methods felt like the closer relative. He also points to a concrete efficiency finding: in May 2023, researchers Gulrajani and Hashimoto quantified that a likelihood-based continuous diffusion language model, Plaid-1B, was 64 times less training-efficient than a comparable baseline. That gap looked especially damning at a time when the field was fixated on the compute-versus-performance tradeoff popularized by Chinchilla, just months after the first LLaMA model, released in February 2023, had begun arguing that inference cost deserved equal weight. Dieleman then lays out the mechanics of adapting continuous diffusion to discrete language data, work that matters again if the approach is regaining ground. Because a continuous corruption process needs a vector space to operate in, the text tokens first need to be embedded as continuous vectors before noise is added, reversing the usual order in which discrete methods embed after corrupting. He surveys three ways researchers have built that embedding space: explicit encodings such as one-hot vectors or, for large vocabularies, compact binary patterns; embeddings pretrained by another model, whether autoregressive or bidirectional, and optionally made contextual; and embeddings learned jointly with the denoising model itself, which Dieleman calls the most natural approach because it mirrors how autoregressive and discrete diffusion models already learn their own embeddings.

Key facts

  • Discrete diffusion language models (multinomial diffusion, D3PM, SUNDAE) came first, in 2021; continuous diffusion, which embeds tokens as vectors before applying Gaussian noise, followed in 2022 starting with Diffusion-LM.
  • A diagram from a 2025 survey paper shows that virtually all new diffusion-language-model research after 2023 shifted to discrete methods, an event Dieleman calls the continuous approach's "extinction."
  • In May 2023, Gulrajani and Hashimoto quantified that continuous diffusion model Plaid-1B was 64 times less training-efficient than a comparable autoregressive baseline.
  • Dieleman, who co-authored two continuous-diffusion papers (SED and CDCD), later moved into building image and video diffusion models, including Imagen, Veo, Nano Banana and Omni.
  • Building a continuous diffusion language model requires embedding discrete tokens into a continuous space first; Dieleman lays out three strategies for that: explicit encodings, embeddings pretrained by another model, or embeddings learned jointly with the denoiser.

Why it matters

Autoregressive generation, one token predicted after another, has become the default assumption behind large language models, but it is not the only way to build a generative model for text. Dieleman's piece is a case for taking the alternative seriously again: continuous diffusion can represent uncertainty at the level of individual tokens and draws on a large toolbox of sampling and distillation tricks developed for image and video diffusion, advantages he argues were given up too quickly when the field consolidated around discrete methods after 2023.

Who it affects

The piece speaks most directly to researchers building diffusion-based alternatives to autoregressive language models, a small but active corner of generative-modeling research. It also reflects Dieleman's own path through that field: he co-authored two continuous-diffusion-for-language papers, then moved on to lead work on image and video diffusion models, including Imagen, Veo, Nano Banana and Omni, before returning to the topic in this retrospective.

How to use it

For a researcher building a continuous diffusion language model, the piece lays out the required ingredients: an embedding strategy that lifts discrete tokens into continuous vectors before noise is applied, a matching loss function, a noise schedule, and self-conditioning, a technique Dieleman says appears in almost every paper on the topic because of its outsized effect on performance. On the embedding choice specifically, he weighs explicit encodings (one-hot vectors, or compact binary patterns for large vocabularies) against embeddings pretrained by another model and embeddings learned jointly with the denoiser.

How solid is it

The one hard, sourced number in the piece, Plaid-1B's 64 times training-efficiency gap versus an autoregressive baseline, comes from a specific paper (Gulrajani and Hashimoto, May 2023) rather than from Dieleman's own testing. The claim that continuous diffusion research went extinct after 2023 rests on a diagram from a separate 2025 survey paper he cites but does not name in detail. Crucially, Dieleman himself labels his explanation for why that shift happened as speculation, not an established finding, and this account does not extend to the specific recent papers he says are now driving a comeback, that part of the post falls outside what the piece covers.

Risks and caveats

Dieleman flags the piece up front as a personal and subjective account, and invites dissenting views in the comments rather than presenting his history as consensus. His reading of why continuous diffusion fell out of favor after 2023, tied to the post-ChatGPT shift toward raw performance and the Plaid-1B efficiency gap, is explicitly his own guess, not a documented cause. He also has a personal stake in the outcome: he says he found continuous diffusion's advantages compelling and was surprised the field abandoned it, a stance worth weighing against the survey evidence he cites.

“I can't help but feel a bit wistful about the apparent extinction of the continuous approach after 2023”

— Sander Dieleman