Glonce

Qwen3.6-27B tests show attention backend and quantization change output

infrastructureThe Glonce editorsHacker News🔥75Glonce rating75Impact60Novelty65Relevance92Surprise60Players40Credibility82
Qwen3.6-27B tests show attention backend and quantization change output

A technical post on the Level1Techs forum, discussed on Hacker News with 223 points and 73 comments, argues that the common complaint that a locally run open model feels dumber than a lab's own hosted version often has nothing to do with different weights. It comes from the inference software and hardware stack computing the same weights differently. The author calls a lab's own first-party hosted version the "reference implementation" and notes that the nightly vLLM container image used for the tests contained 734 packages, 252 of them Python packages installed via uv or pip, each a potential source of divergence between two setups running identical weights.

Test 1 isolates the attention backend. The test model is Qwen3.6-27B, a dense but hybrid model whose 64 layers repeat a pattern of three Gated DeltaNet or linear-attention layers followed by one full-attention layer; only the 16 full-attention layers use a selectable backend. It ran on an RTX PRO 6000 Blackwell GPU at tensor parallelism 1, with BF16 weights and a BF16 KV cache, no quantization, eager execution, CUDA graphs and prefix caching and MTP disabled, and 2k-token chunked prefill. The workload was a real roughly 100,000-token prompt, called "Prompt 2" and drawn from an internal Turnstone lab workstream with multiple tool calls, chosen specifically because it does not appear in any public benchmark or training set. The author ran the same prompt through vLLM's three available full-attention backends, FlashAttention 2, FlashInfer and Triton Attention, using Triton as the baseline, and checked every 32nd token for whether the other backends' greedy top token would have flipped. A repeatability control that reran the same backend multiple times found the stored logits bit-for-bit identical run to run, so the flips seen when switching backends come purely from the different matrix-multiply and addition math each backend's kernel runs during prefill, not from randomness. Early in the prompt all three backends agreed; disagreements showed up later, in clusters tied to what the prompt contained at that point rather than growing smoothly with length.

Test 2 holds the weights at BF16 and quantizes only the KV cache. A full BF16 KV cache stayed fine. An int8 KV cache caused enough top-token flips during tool calls to produce a reproducible tool-calling error, but the model eventually recovered. An int4 KV cache produced the same kind of tool-calling error and did not recover. The section is informally titled around the idea that an LLM's apparent intelligence drops after 40,000 tokens, though that figure appears only in the heading, not as a measured result in the captured text.

Test 3 holds the KV cache at BF16 and compares five weight checkpoints of Qwen3.6-27B, each using a different GEMM and kernel path: the BF16 reference; an official FP8 checkpoint (E4M3 FP8 weights in 128 by 128 blocks with dynamic FP8 activation quantization, lm_head kept in BF16); a community INT8 W8A16 checkpoint (static, symmetric, channel-wise INT8 weights with BF16 activations, GDN and lm_head left unquantized); an NVIDIA NVFP4 checkpoint; and an AWQ W4A16 checkpoint. For the FP8 checkpoint, vLLM automatically disabled DeepGemm because it flagged the checkpoint's E8M0 scale format as accuracy-degrading on this GPU architecture, SM120, and used CUTLASS instead; no calibration dataset was found in the published files. The INT8 checkpoint also shipped with no calibration dataset, using one-shot quantization; the author attributes its unusually good fidelity to combining W8A16 with unquantized GDN projections. The NVFP4 checkpoint is a mixed build: 208 static FP8 W8A8 targets covering the 64 full-attention and 144 GDN projections, and 193 NVFP4 W4A16 targets covering the 192 MLP projections plus lm_head at group size 16; in this upstream-nightly vLLM run, the GPU path was judged to lack native FP4 support, so vLLM ran weight-only FP4 compression through the Marlin kernel rather than native FP4 arithmetic. The captured text cuts off mid-sentence describing the NVFP4 checkpoint's embedded format, before any Test 3 accuracy or KL-divergence results are given.

Throughout, the author uses KL divergence (KLD) to measure how far a quantized model's output probability distribution has moved from the BF16 baseline, framing the approach with a line attributed to a forum user, @wendell: "Math is Math!" A lower KLD means the distribution is closer to the baseline, not automatically that the model is smarter, and KLD is directional, so which distribution is treated as the baseline matters. The author warns that an impossibly low KLD figure on a quantized model's card cannot be interpreted at all unless its author discloses the reference checkpoint, the full runtime environment, the evaluation text, the calibration data, context lengths, sampled positions, KL direction, any vocabulary truncation, and how the measurements were aggregated.

Key facts

  • Switching only the full-attention backend (FlashAttention 2, FlashInfer or Triton Attention) on Qwen3.6-27B running on vLLM flips the model's greedy top token later in a real roughly 100,000-token prompt, even though repeated runs of the same backend produce bit-for-bit identical logits.
  • Quantizing only the KV cache to int8 causes tool-calling token flips that the model recovers from; int4 causes a reproducible tool-calling failure that does not recover, while a full BF16 KV cache stays fine.
  • The nightly vLLM container image used for the tests contains 734 packages, 252 of them Python, each a potential source of divergence between two setups running identical model weights.
  • Comparing five Qwen3.6-27B weight checkpoints (BF16, official FP8, a community INT8 W8A16 build, NVIDIA NVFP4 and AWQ W4A16), the FP8 checkpoint makes vLLM auto-disable DeepGemm and fall back to CUTLASS because its E8M0 scale format is flagged as accuracy-degrading on the test GPU, and neither the FP8 nor the INT8 checkpoint ships a calibration dataset.
  • The captured article text stops mid-sentence during the Test 3 checkpoint description, before any accuracy or KL-divergence numbers comparing the five weight-quantization schemes are given.

Why it matters

The post gives a mechanistic account of a gap local-LLM users routinely notice but rarely diagnose: a downloaded model performing worse than a lab's own hosted version, even when the published weights are identical. By isolating attention backend, KV-cache quantization and weight quantization one at a time, and by running a repeatability control that found bit-for-bit identical logits across repeated runs of the same backend, the author shows the divergence comes from the matrix-multiply math each software and hardware path runs, not from randomness, sampling, or the weights themselves.

Who it affects

Home-lab and self-hosted LLM users who mix GPU generations or switch inference engines; anyone serving open-weight models through vLLM or a similar stack and comparing the results to a lab's hosted reference; and anyone deciding whether to trust a low KL-divergence claim printed on a quantized model's card.

How to use it

Evaluate a local setup against benchmarks and workloads that actually resemble the target use, including long-context and tool-calling tasks, rather than a handful of zero-shot prompts at temperature zero. Match a model card's specified sampler settings and chat template exactly; the author notes that a wrong sampling temperature is a common reason a Qwen model gets stuck looping inside its own think output. For long-context, tool-calling workloads, avoid int4 KV-cache quantization, since it produced a tool-calling failure that did not recover while int8 eventually did. Treat a low KLD figure on a quantized model's card as uninterpretable unless its author discloses the reference checkpoint, full runtime environment, evaluation text, calibration data, context lengths, sampled positions, KL direction, vocabulary truncation and aggregation method.

How solid is it

This is a single, detailed forum post on Level1Techs, not a peer-reviewed study; the captured text gives no byline, is written in first person throughout, and the Hacker News submitter's handle should not be read as the author. The methodology is unusually careful for a forum post: it changes one variable at a time (backend only, then KV cache only, then weights only), captures full-vocabulary logits, and includes a same-backend repeatability control that found bit-for-bit identical logits across repeated runs, which rules out simple run-to-run noise as the source of the flips. The discussion drew 223 points and 73 comments on Hacker News. All results are demonstrated on one model, Qwen3.6-27B, and largely one GPU, an RTX PRO 6000 Blackwell; the captured text cuts off mid-sentence during the Test 3 checkpoint description, before any Test 3 results appear.

Risks and caveats

The findings are shown on one hybrid dense model and mostly one GPU; whether the same pattern holds on other model architectures, GPUs or inference engines is not established in the text as captured. The framing that an LLM's apparent intelligence "drops like a rock after 40k tokens" comes from an informal section heading, not a stated benchmark number. Because the article is cut off before Test 3's results and before any closing synthesis, the piece as captured does not state which weight-quantization scheme is safest overall, only how each checkpoint is built and which kernels it runs on.

“Math is Math!”

— @wendell, quoted in the article