LLaDA MoE v2 nears Qwen3 with about 65% as many pretraining tokens

Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, but how a Mixture-of-Experts (MoE) architecture scales inside a diffusion model had remained poorly understood. The researchers systematically examined how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, and found that the trends differ quantitatively from the scaling trends already reported for AR models.

On optimization, they found that as compute increases, the optimal nominal batch size grows faster and the optimal learning rate decays more rapidly than in AR scaling. On the balance between data and model size, an equal-compute (IsoFLOP) comparison revealed a slight tilt toward data: the optimal token budget grows faster than the optimal amount of activated model-side computation. On MoE architecture itself, larger scales increasingly favor larger pools of experts at a fixed activated capacity, moderate expert granularity keeps working well across scales, and the preferred share of activated capacity given to shared experts stays stable as scale increases.

Guided by these findings, the researchers trained LLaDA MoE v2, a 30B-A3B Mixture-of-Experts diffusion language model (30 billion total parameters, about 3 billion active per token), from scratch on 23.5 trillion tokens, about 65% as many pretraining tokens as Qwen3 used. LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of the eight reasoning and coding benchmarks tested and remains close to Qwen3 on several tasks.

The researchers present these results as evidence for practical scaling laws and design principles specific to MoE diffusion language models, rather than assuming that scaling recipes built for autoregressive models carry over unchanged.

Key facts

LLaDA MoE v2 is a 30B-A3B Mixture-of-Experts diffusion language model, trained from scratch on 23.5 trillion tokens.
That is about 65% as many pretraining tokens as Qwen3 used, yet LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning and coding benchmarks.
After supervised fine-tuning alone, LLaDA MoE v2 outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks.
The researchers found that for MoE diffusion models, the optimal batch size grows faster and the optimal learning rate decays more rapidly with compute than the scaling trends reported for autoregressive models.
An equal-compute (IsoFLOP) analysis showed a slight data-side tilt, with the optimal token budget growing faster than the optimal activated compute, while larger scales favor larger expert pools at a fixed activated capacity.

Why it matters

Diffusion language models are an alternative to the autoregressive language modeling that most large language models use, but how a Mixture-of-Experts architecture scales inside a diffusion model had not been characterized before this work. By systematically studying optimization, the balance between data and compute, and architecture choices, the researchers produced scaling laws specific to MoE diffusion models rather than relying on rules built for autoregressive ones. LLaDA MoE v2 is the practical proof: a model trained from these findings that approaches Qwen3 on several benchmarks using about 65% as many pretraining tokens.

Who it affects

Researchers and engineers building or scaling diffusion language models, especially those combining diffusion with Mixture-of-Experts architectures, gain a tested set of scaling guidelines instead of having to rediscover them by trial and error. Anyone deciding whether to move pretraining away from the standard autoregressive approach, or trying to cut the number of tokens needed to reach a target performance level, is the direct audience for these findings.

How to use it

There is no product here to install: this is a research paper describing scaling laws and a resulting model checkpoint. The practical design guidance it offers: at larger scale, batch size grows faster and learning rate decays faster with compute than autoregressive conventions suggest; the data budget tilts slightly higher relative to activated compute; larger expert pools work better at larger scale while expert granularity stays moderate; and the share of capacity given to shared experts stays roughly constant across scales. The abstract does not state whether LLaDA MoE v2's weights are released, under what license, or on what timeline, so practical access to the model itself is not established by this description.

How solid is it

The findings rest on a systematic empirical study across optimization, data and compute allocation, and MoE architecture, and the authors test their own conclusions directly by training a 30B-A3B model end to end and comparing it against Qwen3 and SDAR Chat. The abstract does not name the authors, any institutional affiliation, or a publication date, and it gives no exact figures behind the qualitative 'grows faster' and 'decays more rapidly' language or the specific benchmarks used, so several details needed to judge the evidence independently are not available from this description. The benchmark comparisons are the authors' own reported results, with no indication in the abstract of independent replication.

Risks and caveats

LLaDA MoE v2 approaches Qwen3 rather than matching or beating it, and only on several unnamed benchmarks, which leaves the rest of its performance profile unclear. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight tested benchmarks, meaning there is at least one benchmark where it does not. The abstract gives no figures for the training cost, hardware, or time behind the 23.5-trillion-token run, and it says nothing about whether the model or its weights will be released. The scaling trends the researchers describe are specific to the architectures and scale ranges they tested and may not hold outside that range.