Frequency-aware remasking lifts LLaDA and Dream on MATH-500 and HumanEval

Frequency-aware remasking lifts LLaDA and Dream on MATH-500 and HumanEval

Diffusion language models (DLMs) generate text by iterative denoising, and the choice of which tokens to commit and which to remask at each step strongly affects output quality. The paper argues that existing certainty-aware strategies, which rely on instantaneous confidence or entropy, use local signals that can be misleading on difficult reasoning problems.

The authors frame DLM test-time scaling as adaptive allocation of both inference compute and decoding strategies. Their approach has two stages. A fast base sampler first solves the easy questions. Frequency-aware remasking is then applied only to the remaining hard ones.

The remasking step tracks how often each token's prediction changes across denoising steps and preferentially commits the tokens that change frequently. The stated aim is to push samples toward alternative reasoning paths instead of collapsing onto the model's preferred trajectory.

The experiments use LLaDA-8B-Instruct and Dream-7B-Instruct. The paper reports consistent improvements over a strong adaptive allocation baseline. On MATH-500, absolute accuracy gains reach up to 3.73 percentage points for LLaDA and 7.33 percentage points for Dream. On HumanEval, absolute coverage gains reach up to 11.39 percentage points for LLaDA and 10.98 percentage points for Dream.

The paper also looks at questions the base certainty-aware strategy left unsolved after 100 samples. On those, frequency-aware remasking recovers solutions for 3.75% to 18.75% of examples.

Key facts

  • The method is two-stage: a fast base sampler handles easy questions, and frequency-aware remasking is applied to the remaining hard ones.
  • Frequency-aware remasking tracks how often each token's prediction changes across denoising steps and preferentially commits frequently changing tokens.
  • On MATH-500, absolute accuracy gains over a strong adaptive allocation baseline reach up to 3.73 percentage points (LLaDA-8B-Instruct) and 7.33 (Dream-7B-Instruct).
  • On HumanEval, absolute coverage gains reach up to 11.39 percentage points (LLaDA) and 10.98 (Dream).
  • On questions the base strategy left unsolved after 100 samples, the method recovers solutions for 3.75% to 18.75% of examples.

Why it matters

Diffusion language models are an alternative to left-to-right text generation, and how to spend extra compute at inference time is still an open question for them. This paper targets one specific weakness: decoding rules based on a token's current confidence or entropy can mislead on hard reasoning problems. Its answer is to look at how stable each token's prediction is over the denoising steps, and to spend the extra effort only on questions the cheap sampler cannot solve.

Who it affects

Mainly researchers and engineers working on diffusion language models and on inference-time compute allocation. The experiments cover two open instruction-tuned models, LLaDA-8B-Instruct and Dream-7B-Instruct, so the results speak most directly to people using those models.

How to use it

The paper describes a recipe rather than a product. Run a fast base sampler first; for questions it fails to solve, switch to sampling with frequency-aware remasking, which tracks how often each token's prediction changes across denoising steps and commits the frequently changing tokens first. The abstract does not say whether code is released.

How solid is it

The results come from the paper's own abstract on Hugging Face Papers and are stated as maxima: the gains are "up to" the figures given, not averages. The abstract names no authors or institutions and gives no submission date. It does not state which baseline is meant by "strong adaptive allocation baseline" beyond that phrase, and it does not say how many examples sit behind the 3.75% to 18.75% range. Baseline absolute accuracy and coverage levels are not given, so the point gains cannot be turned into relative improvements.

Risks and caveats

Evidence covers only MATH-500 and HumanEval and only two models, so nothing is shown for other benchmarks or model families. The abstract reports no compute cost, latency or overhead for the method, and does not state the sample budgets or configurations behind each maximum gain. Percentage-point gains and the 3.75% to 18.75% recovery range are different measures and should not be read as one number.

“these local signals can be misleading on difficult reasoning problems”

— Paper abstract, Hugging Face Papers