Glonce

Qwen3.8-27B's Gated DeltaNet layers quantize to 4-bit NVFP4 without a performance hit

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact70Novelty72Relevance95Surprise55Players30Credibility85
Qwen3.8-27B's Gated DeltaNet layers quantize to 4-bit NVFP4 without a performance hit

Hybrid large language models combine ordinary softmax attention with linear-attention layers such as Gated DeltaNet (GDN), whose recurrent state summarizes the entire context in a fixed size rather than growing with it. Qwen3.8-27B is one such hybrid model, built from 48 GDN layers and 16 attention layers. Earlier community efforts to quantize it to 4 bits had left the GDN block, and especially its decay and write-strength gates, at 8-bit or 16-bit precision, on the intuition that errors inside a recurrence accumulate over long contexts and would otherwise degrade the model.

A new paper tests that intuition directly by building a recipe called Minima, which quantizes all 496 linear layers of Qwen3.8-27B, the GDN layers included, to NVFP4 W4A4: 4-bit weights and 4-bit activations in Nvidia's NVFP4 format. Minima is evaluated against the original BF16 model on perplexity at 4K and 32K context lengths, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval out to 64K context.

Minima matches BF16 within seed noise, with a 5-task average score difference of -0.52 (the paper does not specify which five of the tasks make up that average). It is also the smallest checkpoint among the recipes compared, at 17.5 GiB, and the fastest at prefill, 14 to 19 percent quicker. Its perplexity gap against BF16 at 32K context shrinks as position in the sequence advances, meaning the quantized model tracks the full-precision one more closely later in a long context, not less closely: the opposite of what the accumulating-error intuition would predict.

A four-part mechanism study explains the result. First, NVFP4's block scaling operates on blocks of 16 elements, which localizes the extreme outlier values found in the residual stream and equalizes activation error across different layer roles instead of letting it concentrate in one place. Second, the gate projections, the component assumed most fragile, turn out to be the least sensitive to quantization: their softplus/exponential and sigmoid parameterizations compress about 11 percent numerical error at the matrix-multiply level down to about 2 percent error at the model's output. Third, the delta-rule recurrence that GDN uses does not let injected noise build up: noise sits at a flat plateau over 32K tokens, and a one-time perturbation to the state is forgotten within hundreds of steps, because every new write overwrites the state along the current key direction. Fourth, as a consequence, the per-token cost of quantization washes out as the context grows rather than compounding, contrary to what motivated keeping GDN at higher precision in the first place.

The paper also fixes an engineering problem: when NVFP4 checkpoints calibrated separately for each module are served by inference kernels that fuse several modules into a single matrix-multiply operation, a global-scale mismatch results, which the authors repair. They further show that a calibrated FP8 KV-cache adds no performance cost. Their overall conclusion is a practical recipe, quantize everything and ship calibrated KV-cache scales, backed by a mechanistic account of why the recurrent half of a hybrid LLM turns out to be the easy half to quantize. The quantized checkpoint is published on Hugging Face as minima-ai/mnma_qwen3.8_27b_nvfp4.

Key facts

  • Minima quantizes all 496 linear layers of the hybrid LLM Qwen3.8-27B (48 Gated DeltaNet layers, 16 attention layers) to 4-bit NVFP4 W4A4, including the recurrent GDN block that earlier 4-bit quantizations had left at 8-bit or 16-bit precision.
  • Across perplexity (4K/32K), MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench and RULER retrieval to 64K, Minima matches BF16 within seed noise, with a 5-task average score difference of -0.52 (the paper does not say which five tasks).
  • Minima is the smallest checkpoint among the recipes compared, at 17.5 GiB, and the fastest at prefill, 14 to 19 percent quicker.
  • The gate projections, assumed to be the most fragile component, turn out to be the least sensitive to quantization: about 11 percent numerical error at the matrix-multiply level compresses to about 2 percent error at the model's output.
  • The delta-rule recurrence does not let injected noise build up over a 32K-token context and forgets a one-time state perturbation within hundreds of steps, because each write overwrites the state along the current key direction.

Why it matters

Hybrid architectures that mix softmax attention with a linear-attention component like Gated DeltaNet are one of the main routes to cheaper long-context inference, because the recurrent part holds a fixed-size state instead of a cache that grows with the sequence. The catch has been an assumption that the recurrent component, and especially its gating math, needs more numeric precision than the rest of the network, which limits how far the whole model can be compressed. This paper tests that assumption directly on Qwen3.8-27B and finds it does not hold: quantizing the Gated DeltaNet layers to 4-bit NVFP4 alongside everything else still matches BF16 performance, while cutting checkpoint size and speeding up prefill. The authors frame the result as a general point about hybrid LLMs, that the recurrent half is the easy half to quantize, not a narrow one about this single model.

Who it affects

Teams that build or serve hybrid attention/linear-attention models, where VRAM and prefill latency are the binding costs of running a large model, are the direct audience. Infrastructure and inference engineers who maintain quantization and serving kernels are affected by the paper's second contribution, a fix for a global-scale mismatch that appears specifically when per-module-calibrated NVFP4 checkpoints are served by kernels that fuse several modules into one matrix-multiply operation. Researchers working on quantization and on linear-attention or state-space architectures more broadly are the other clear audience, since the mechanism study gives a concrete, testable explanation rather than only a benchmark result.

How to use it

The paper releases the quantized model itself on Hugging Face as minima-ai/mnma_qwen3.8_27b_nvfp4, so anyone running Qwen3.8-27B can adopt the checkpoint directly. The underlying recipe is stated plainly: quantize every linear layer, including the Gated DeltaNet layers, to NVFP4 W4A4, and pair it with a calibrated FP8 KV-cache, which the paper reports costs no performance. Teams serving the model through kernels that fuse per-module-calibrated NVFP4 into a single matrix-multiply operation need the paper's global-scale-mismatch fix as well, or they risk the same mismatch it describes. No price or license terms for the checkpoint are given in the text.

How solid is it

The paper does not name its authors or institution, referring to itself throughout only as "we," and states no publication venue or date, so there is no track record to weigh independently of the result itself. The evaluation is reasonably broad for a single paper: perplexity at two context lengths, a 5-task average drawn from the listed downstream benchmarks (the paper does not say which five), and a four-part mechanism study that ties the benchmark result to a specific explanation rather than stopping at the numbers. The recipes Minima is compared against are described only in relative terms, as bigger or slower, with no absolute sizes or speeds given in the text, so the size of its advantage over any one named alternative cannot be checked directly. As a single, self-reported study, independent replication would strengthen the claim.

Risks and caveats

The result is demonstrated on one model, Qwen3.8-27B, and one quantization scheme, NVFP4 W4A4; whether it generalizes to other hybrid architectures, other linear-attention mechanisms, or other model sizes is not established here. The headline 5-task average of -0.52 is not tied to named tasks, which limits how precisely the claim of matching BF16 can be checked. Because the comparison recipes are not given absolute sizes or speeds in the text, the 17.5 GiB figure and the 14 to 19 percent prefill advantage are only relative to unnamed alternatives. Deploying the recipe through kernels that fuse per-module-calibrated NVFP4 modules into one matrix-multiply operation requires the paper's global-scale-mismatch fix; skipping it reintroduces the exact problem the paper describes.

“the recurrent half of a hybrid LLM is the easy half to quantize”

— the paper