Glonce

HOPE prunes MoE experts as partners and beats REAP at 50% pruning

researchThe Glonce editorsHugging Face🔥62Glonce rating62Impact50Novelty60Relevance95Surprise30Players50Credibility75
HOPE prunes MoE experts as partners and beats REAP at 50% pruning

Mixture-of-Experts (MoE) models carry a memory cost: every expert MLP has to sit in memory at inference time, though only a few are used for any token. The post's example is Qwen3.5-122B, which has 256 experts in each of its 48 MoE layers. Expert MLPs take up 95% of its roughly 250 GB parameter footprint, yet only 8 experts per layer are active for a given token.

Expert pruning, permanently deleting the least useful experts, is presented as a direct fix. According to the authors, it avoids the latency problems of dynamically loading and off-loading experts, needs no training, and composes with quantization. The hard part is deciding which experts to remove. Existing methods give each expert its own scalar importance score from a calibration dataset and cut the lowest-scoring experts in each layer independently. These are first-order methods: they assume every expert contributes independently. The authors argue that in reality experts form cooperative groups that are consistently co-activated, and that ignoring this leads to poor pruning, especially at aggressive rates. Their example: removing expert A may look harmless alone, but if expert B, which cooperates with and relies on A, is highly important, deleting A breaks the pair.

Their newest paper derives HOPE (Higher-Order Pruning of Experts), a second-order method that accounts for pairwise expert interactions. When experts are deleted, tokens are rerouted to the experts with the next-highest gate weights, and the resulting change in layer output is the pruning error. Minimizing that error directly is intractable, because it depends on which replacement experts get chosen, which depends on the prune-set itself. HOPE instead minimizes a surrogate that upper-bounds the dominant component of the error. Because the surrogate is a squared sum, expanding it produces cross-terms between pairs of experts, the interaction terms first-order methods drop. The problem becomes a quadratic program over an "F-matrix" collected on the calibration data. The diagonal measures individual expert importance; the off-diagonal entries measure pairwise cooperative contribution. The program is solved with a continuous relaxation, then rounded to binary choices. It is non-convex but a standard QP solver handles it in 1 to 2 seconds per layer, and the authors say the extra overhead is negligible next to the calibration pass, which dominates.

HOPE is closely tied to REAP (Router-weighted Expert Activation Pruning), a state-of-the-art first-order method. The F-matrix diagonal recovers REAP's expert rankings almost perfectly, so the authors attribute any gain over REAP to the off-diagonal interaction terms.

The evaluation used 3 models (Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air) and 2 calibration datasets (Evol-CodeAlpaca and SWE-Bench Verified trajectories). Pruned models were tested on the Tulu3 suite and on SWE-Bench Pro, an agentic coding benchmark. HOPE was compared with 4 first-order baselines (REAP, MAN, EAN and Frequency) at pruning rates from 10% to 50%. Across all conditions HOPE had the best average rank, 2.07 out of 5 methods, with top-1 finishes in 39% of conditions and top-2 finishes in 67%. The post says it beats every other baseline head to head in the majority of conditions.

The advantage grows with the pruning rate. At 50% pruning HOPE's average rank is 1.58, against 2.42 for the next-best method, REAP. The post reports gains of up to +6.1% on agentic coding tasks over the next-best method at that rate, without saying whether that is percentage points or relative. The clearest example is Qwen3.5-35B-A3B at 50% pruning on SWE-Bench Pro: HOPE scored 42.3, matching the unpruned model's 41.8, while REAP fell to 36.2, which the post calls a 13% regression. The authors suggest agentic tasks likely exercise diverse expert combinations, where preserving cooperative structure matters most.

Two further observations. Pruned models can occasionally match or exceed the unpruned baseline, which the authors say is likely because removing low-relevance experts reduces noise; this is not unique to HOPE. And code-focused calibration with any pruning method, HOPE included, did not severely degrade non-coding benchmarks, with HOPE performing well across Tulu3 regardless of calibration set. Looking at experts where HOPE and REAP disagree, the experts only HOPE keeps have higher average F-values with other retained experts than those only REAP keeps. So HOPE gives up some individually high-scoring experts to preserve cooperative clusters.

The authors released a basic package with three command-line steps. "hope calibrate" runs one forward pass over calibration data (agentic rollouts or single-turn traces) using HuggingFace transformers hooks, and stores the F-matrix in an HDF5 file. "hope solve" takes a pruning fraction such as 0.50, solves the quadratic program, and writes the prune-set as JSON. "hope prune" builds the smaller checkpoint. The result is a standard HuggingFace checkpoint, smaller on disk and in memory, with no changes needed to vLLM or SGLang model definitions. The package has been tested on Qwen3, Qwen3-Next, Qwen3.5 and GLM-4.5, and a more compact Python API is also provided.

Key facts

  • HOPE (Higher-Order Pruning of Experts) is a training-free, second-order method for pruning experts in MoE LLMs; it models pairwise expert interactions through a quadratic program, with a solve time of 1 to 2 seconds per layer.
  • Across 3 models, 2 calibration sets and pruning rates from 10% to 50%, HOPE had the best average rank against 4 first-order baselines: 2.07 out of 5, with top-1 finishes in 39% of conditions and top-2 in 67%.
  • At 50% pruning HOPE's average rank is 1.58 versus 2.42 for the next-best method, REAP; the post reports gains of up to +6.1% on agentic coding tasks.
  • On Qwen3.5-35B-A3B at 50% pruning, HOPE scored 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8, while REAP dropped to 36.2.
  • The released package runs calibrate, solve and prune steps and outputs a standard HuggingFace checkpoint that needs no changes to vLLM or SGLang model definitions.

Why it matters

In MoE models nearly all parameters are expert MLPs: 95% of Qwen3.5-122B's roughly 250 GB footprint, with only 8 experts per layer used per token. Pruning experts is a direct way to cut that memory, and the authors say it needs no training and composes with quantization. HOPE's contribution is to stop scoring experts one by one and to score pairs, which the authors argue matters most at aggressive pruning rates. They also note that 50% pruning is the most deployment-relevant setting, since the savings are large enough to leave room for longer contexts, deeper reasoning and more tool calls.

Who it affects

Anyone who runs MoE models for inference and is limited by memory, since pruned checkpoints are smaller on disk and in memory. The package was tested on Qwen3, Qwen3-Next, Qwen3.5 and GLM-4.5 families. People serving coding agents are the most direct audience, because the biggest reported gains are on SWE-Bench Pro.

How to use it

The package has three steps. First, "hope calibrate" with a model path and a prompts file, producing an HDF5 file of observations. Second, "hope solve" with that file and a prune fraction (the example uses 0.50), producing a JSON prune-set. Third, "hope prune" with the model path and prune-set, producing the smaller checkpoint. Calibration data can be agentic rollouts or single-turn traces. The output loads like any HuggingFace checkpoint, and the post says vLLM and SGLang model definitions need no changes. A Python API with calibrate, solve and prune_model functions is also provided.

How solid is it

The numbers come from the authors' own evaluation: 3 models, 2 calibration datasets, 4 first-order baselines, pruning rates from 10% to 50%, on the Tulu3 suite and SWE-Bench Pro. The post is backed by a paper, with the full derivation in its Appendix A. The headline average rank (2.07 out of 5) is a ranking, not a score gap. No independent replication or third-party reaction is reported. The visible text names no authors, institutions or paper title or link.

Risks and caveats

The +6.1% gain is not specified as percentage points or relative. The SWE-Bench Pro example is a single model at one pruning rate, and no SWE-Bench Pro or other benchmark numbers are given for Qwen3.5-122B or GLM-4.5-Air in the visible text. HOPE does not win everywhere: it ranks first in 39% of conditions, and its advantage is smaller at lower pruning rates. The explanations that agentic tasks "likely" exercise diverse expert combinations and that pruned models matching the baseline is "likely" noise reduction are hedged by the authors and not demonstrated. No concrete memory or disk size figures for the pruned models are given. The quadratic program is non-convex and solved via relaxation and rounding.

“HOPE sacrifices some individually high-scoring experts in favor of preserving cooperative clusters”

— HOPE blog post on Hugging Face