Paper: MoE channel selection needs no calibration data, top-k beats thresholds

Mixture-of-Experts (MoE) models are bottlenecked by loading routed-expert weights from memory during interactive decoding at small batch sizes. Most intermediate channels in a routed expert contribute little to any single token, so channel selectors predict which channels matter and read only their weights. The usual approach keeps channels above a threshold calibrated on offline data.

The paper argues that this calibration is unnecessary, and makes two points. First, a calibrated threshold does not fix how much a selector reads, because the number of channels passing it varies across inputs. As an example, Prox with thresholds calibrated for an 18% read budget actually reads only 12.9% of weights on average across the authors' benchmarks. Second, top-k keeps each token's k highest-scoring channels and needs no calibration data. At the calibrated budget, a selector's top-k version matches or beats its threshold version on the same score in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks. The top-k versions also avoid the repetition-loop collapse that calibrated thresholds cause on Gemma 4.

The authors conclude that calibrating a threshold provides neither of the two things selectors use it for: a known read budget and a consistent accuracy gain at that budget.

They also introduce a reader built entirely from the model's weights. It screens channels with a ternary sketch and recovers the skipped ones with a low-rank map. Across three post-trained MoE models, the reader retains 98.1% of full-model accuracy at 17.8% of routed-expert reads. Their Triton kernels run MoE blocks 1.60 to 1.90x faster than optimized vLLM at batch one, and up to 2.69x faster at larger batches, on the same three models.

Key facts

Calibrated thresholds do not fix the read budget: Prox calibrated for an 18% budget actually read 12.9% of weights on average across the authors' benchmarks.
A selector's top-k version matched or beat its threshold version in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks.
Top-k versions avoid the repetition-loop collapse that calibrated thresholds cause on Gemma 4.
The authors' weights-only reader keeps 98.1% of full-model accuracy at 17.8% of routed-expert reads across three post-trained MoE models.
Triton kernels run MoE blocks 1.60 to 1.90x faster than optimized vLLM at batch one, and up to 2.69x at larger batches.

Why it matters

At small batch sizes, MoE decoding is limited by how fast routed-expert weights can be loaded from memory. Selectors that read only the channels that matter attack that bottleneck, and they have normally relied on thresholds calibrated on offline data. The paper's claim is that this calibration step can be dropped: per-token top-k gives a fixed read budget and matches or beats the calibrated version in most of the authors' comparisons.

Who it affects

The work is aimed at people building and serving MoE models for interactive, small-batch decoding, and at authors of channel-selection methods who use calibrated thresholds. The paper does not name the three post-trained MoE models it tests, other than mentioning Gemma 4 as where threshold collapse occurs.

How to use it

The practical idea is to replace a calibrated threshold with top-k: keep each token's k highest-scoring channels, so no calibration data is needed and the read budget is set directly. The authors also describe a reader that works from the model's weights alone, using a ternary sketch to screen channels and a low-rank map to recover skipped ones, paired with Triton kernels. The abstract does not say whether the kernels or code are released.

How solid is it

The evidence is the authors' own benchmarks. The top-k result rests on 43 comparisons at 13% and 18% budgets, of which 40 favour or tie top-k, and all 23 generation-task comparisons do. The accuracy and speed figures (98.1%, 17.8%, 1.60 to 1.90x, up to 2.69x) are aggregates over three models; per-model results are not given in the abstract. The batch sizes behind "larger batches" are not specified. This is a paper summary and has not been independently verified.

Risks and caveats

Top-k matched or beat thresholds in 40 of 43 comparisons, so it did not do so in 3. The figures are averages over unnamed benchmarks and models, so results on other models or tasks may differ. The abstract does not say which selectors besides Prox were compared. The speedups are measured against optimized vLLM on the authors' three models, and the abstract gives no detail on hardware or on the larger batch sizes.

“Calibrating a threshold thus provides neither of the two things selectors use it for: a known read budget and a consistent accuracy gain at that budget.”

— From the paper's abstract, Hugging Face papers