Qwen3.6-35B-A3B tolerates aggressive pruning in its late MoE layers

Mixture-of-experts (MoE) architectures let large language models scale while keeping inference sparse and efficient, but how much each individual MoE layer actually contributes has not been well characterized, particularly for compression purposes. To address that gap, researchers ran a systematic layer-wise sensitivity analysis on Qwen3.6-35B-A3B, an MoE model with 40 MoE layers, 256 experts per layer and top-8 routing (eight active experts per token, 10,240 experts in total). The method was magnitude-based expert masking: zeroing out low-magnitude experts and measuring the effect on output quality on the XLCoST cross-lingual code translation benchmark. The study ran in multiple phases at 100, 300 and 500 prompt evaluation scales, using three H100 GPU servers.

The central finding is that layer sensitivity is strongly depth-dependent. Early layers (0-9) and middle layers (10-29) are highly fragile to expert masking: removing low-magnitude experts there degrades output quality sharply. Late layers (30-39), and especially very-late layers (35-39), tolerate aggressive masking of low-magnitude experts far better.

The numbers back this up. At the 300-prompt scale, masking 30% of experts uniformly across all layers ("flat all-layer masking") retains only 150 of 300 outputs rated Good or Similar to the unmasked baseline. Late-focused masking policies, which concentrate masking in the tolerant late layers instead of spreading it evenly, retain 249 to 255 of 300 Good+Similar outputs while masking a comparable 640 to 1,145 experts.

On a separate, later 500-prompt held-out validation slice, a narrow policy that masks only layers 35-39 at a 50% ratio produced the strongest quality-to-masked-expert tradeoff among the candidates tested: it retained 419 of 500 Good+Similar outputs while masking just 640 of the model's 10,240 total experts.

The researchers also tested reducing the routing width itself, from the model's baseline of 8 active experts per token down to 6. On a 100-prompt probe this produced a large wall-clock speedup with no loss of Good+Similar outputs, though no specific percentage or time figure is given for that speedup. The authors note this routing-width reduction does not yet compose cleanly with aggressive expert masking, without detailing why.

The authors frame the results as an empirical foundation for depth-aware MoE expert masking, and as a practical path toward further techniques: physical weight surgery, activation-based expert scoring and training-based recovery.

Key facts

Qwen3.6-35B-A3B has 40 MoE layers, 256 experts per layer and top-8 routing; the study used magnitude-based expert masking on the XLCoST benchmark across 100/300/500-prompt scales on three H100 servers.
Early layers (0-9) and middle layers (10-29) are highly fragile to expert masking; late layers (30-39), especially very-late layers (35-39), tolerate aggressive masking.
At 300-prompt scale, flat 30% masking across all layers retains only 150/300 Good+Similar outputs, versus 249-255/300 for late-focused policies masking 640-1,145 experts.
On a 500-prompt validation slice, a narrow policy masking only layers 35-39 at 50% retained 419/500 Good+Similar outputs while masking just 640 of the model's 10,240 total experts.
Reducing routing width from top-8 to top-6 gave a large wall-clock speedup with no Good+Similar loss on a 100-prompt probe, but does not yet compose cleanly with aggressive expert masking.

Why it matters

MoE models scale efficiently through sparse activation, but this paper argues that which layers actually matter for compression has not been well characterized. The result reframes expert pruning as a depth-aware problem rather than a uniform one: late layers, and especially very-late layers 35-39, can absorb far more aggressive masking than early or middle layers without the same quality loss. Depth-aware masking policies preserved substantially more Good+Similar output quality than flat, uniform masking at a comparable number of masked experts, giving compression work a concrete empirical basis rather than a guess.

Who it affects

Teams running large MoE models like Qwen3.6-35B-A3B (40 layers, 256 experts per layer, 10,240 experts total) who want to cut inference cost through expert pruning or reduced routing width, and researchers working on MoE compression, activation-based expert scoring, or training-based recovery methods that build on this kind of sensitivity map.

How to use it

The strongest tradeoff the authors tested is specific: masking only layers 35-39 at a 50% ratio retained 419 of 500 Good+Similar outputs on a held-out validation slice while removing just 640 of the model's 10,240 experts, well ahead of flat all-layer masking at a similar budget. Separately, cutting top-k routing from 8 to 6 active experts per token gave a large wall-clock speedup on a small, 100-prompt probe with no quality loss, but the paper does not yet show it combining cleanly with expert masking. No pricing, licensing or code-release details appear in the source.

How solid is it

The study spans three evaluation scales (100, 300 and 500 prompts), including a held-out 500-prompt validation slice run separately from the earlier phases, executed across three H100 GPU servers and scored against the XLCoST cross-lingual code translation benchmark using Good/Similar output categories. The extracted text gives no author names, institutional affiliations or publication date, so provenance beyond the arXiv listing cannot be confirmed here.

Risks and caveats

The findings are specific to one model (Qwen3.6-35B-A3B), one benchmark (XLCoST code translation) and one masking method (magnitude-based); how well the depth-dependent pattern generalizes to other MoE architectures or tasks is untested. The routing-width speedup is described only as "large," with no percentage or time figure given, and the authors themselves note it does not yet compose cleanly with aggressive expert masking, without explaining why. The paper describes an "empirical foundation" and a "path toward" further techniques: physical weight surgery, activation-based expert scoring, training-based recovery that are named as future directions rather than demonstrated here.