Glonce

ESRL boosts Qwen3-30B-A3B RL through expert-routing exploration

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact65Novelty72Relevance98Surprise48Players35Credibility87
ESRL boosts Qwen3-30B-A3B RL through expert-routing exploration

Reinforcement learning (RL) has become central to post-training large language models. Recent work on RL for Mixture-of-Experts (MoE) models has mostly focused on optimization stability and training efficiency, treating which experts get selected, the routing, as a fixed part of the model. The authors of a new paper start from a different observation: because routing determines the sparse computation paths that produce a model's output, expert selection is itself an additional source of rollout diversity. Their empirical analysis finds that perturbing expert routing alters a model's output and increases rollout diversity in a way similar to raising the decoding temperature. But perturbing routing directly is risky: it can activate experts that are not suited to the input and substantially degrade the quality of the resulting rollouts.

Motivated by that finding, the authors introduce Expert-Space Exploration Reinforcement Learning (ESRL), an architecture-aware framework that explicitly explores the expert-routing space of MoE models instead of leaving it fixed. ESRL keeps high-confidence experts as anchors and restricts any stochastic routing to a plausible candidate pool, so the model still uses computation paths known to be reliable. The strength of the perturbation is not fixed either; it adapts to the router's entropy, so the framework backs off from over-perturbing when the router is already uncertain. To handle the mismatch that perturbed routing can introduce between the paths used to generate a rollout and the paths assumed during optimization, ESRL records which expert paths were actually used during rollout and replays those same paths when optimizing the policy.

The authors report that ESRL achieves the best performance across MoE backbones with top-K, top-1, and shared-expert routing, and across mathematics, science, and code tasks, without needing additional sampling or computational cost. On Qwen3-30B-A3B specifically, they report ESRL as the best among all the methods they compared it with, improving average Pass@1 and Pass@8 over the GRPO baseline by 3.2 and 4.5 percentage points, respectively; the paper gives only that improvement margin and does not state the absolute Pass@1 or Pass@8 scores behind it. The authors add further analyses of expert utilization and training dynamics, which they say show how exploiting MoE-specific routing structure benefits RL training more broadly.

Key facts

  • ESRL, short for Expert-Space Exploration Reinforcement Learning, is an architecture-aware RL framework that explicitly explores a Mixture-of-Experts (MoE) model's expert-routing space, rather than treating expert selection as fixed during RL post-training.
  • The authors' empirical analysis finds that perturbing expert routing alters model output and increases rollout diversity similar to raising the decoding temperature, but direct perturbation can activate unsuitable experts and substantially degrade rollout quality.
  • ESRL anchors high-confidence experts, restricts stochastic routing to a plausible candidate pool, and adapts perturbation strength to router entropy; it also records the expert paths used during rollout and replays them during policy optimization to correct the resulting routing mismatch.
  • ESRL achieves the best performance across MoE backbones with top-K, top-1, and shared-expert routing, and across mathematics, science, and code tasks, without additional sampling or computational cost.
  • On Qwen3-30B-A3B specifically, ESRL is reported as the best among all compared methods, improving average Pass@1 and Pass@8 over the GRPO baseline by 3.2 and 4.5 percentage points, respectively.

Why it matters

RL post-training for MoE models has generally treated which experts get activated, the routing, as fixed, and focused instead on optimization stability and training efficiency. This paper's starting point is that routing itself is a lever: since it determines the sparse computation paths behind a model's output, perturbing it can increase rollout diversity in a way similar to raising the decoding temperature, according to the authors' empirical analysis. The catch is that doing this carelessly is not free: direct perturbation can activate experts that do not suit the input and substantially degrade rollout quality. ESRL matters because it is offered as a way to capture the diversity benefit of routing perturbation while controlling for that quality cost, and the authors report it improving results without extra sampling or compute.

Who it affects

The paper speaks to researchers and engineers training MoE language models with reinforcement learning, since the reported gains span multiple routing schemes (top-K, top-1, shared-expert) and multiple task domains (mathematics, science, code) rather than one narrow setup. The concrete headline number, the 3.2 and 4.5 percentage-point gains over GRPO, is reported specifically for Qwen3-30B-A3B, so anyone training on a similarly built MoE backbone is the most directly addressed audience. The abstract names no individual authors or institutions, so who built ESRL cannot be established from the text alone.

How to use it

ESRL is a training-time RL technique, not a released tool or product, and the abstract describes it at the level of a method rather than a how-to. Mechanically, it anchors high-confidence experts so the model keeps using computation paths already known to be reliable, and restricts any stochastic exploration to a plausible candidate pool of experts. The perturbation strength adapts to the router's own entropy, so the framework pulls back from over-perturbing when the router is already uncertain. Because perturbed routing during rollout can otherwise mismatch the paths assumed during policy optimization, ESRL records the expert paths actually used in each rollout and replays them when the policy is optimized.

How solid is it

The evidence is the authors' own reported experiments: ESRL is said to beat the compared methods across MoE backbones with different routing schemes and across mathematics, science, and code tasks, and specifically to be the best among all compared methods on Qwen3-30B-A3B, where it improves average Pass@1 and Pass@8 over GRPO by 3.2 and 4.5 percentage points. Those are percentage-point deltas, not the underlying scores: the paper does not give the absolute Pass@1 or Pass@8 numbers for either ESRL or GRPO, so there is no way to tell from the abstract alone how strong either method's raw performance actually is. As of this writing, the paper has drawn five points and two comments on Hugging Face's papers board, so it has not yet attracted much outside scrutiny.

Risks and caveats

Every number here is self-reported by the authors, with no external benchmark or third-party replication cited in the text. The abstract names no individual authors or institutions and gives no submission or publication date, so the paper's provenance cannot be checked from the text alone. It also gives no dataset or benchmark names beyond the general task categories, no training-compute figures, and does not explain what "A3B" denotes in the model name Qwen3-30B-A3B, all of which limits how far the result can be checked or reproduced from the abstract alone.

“perturbing expert routing effectively alters model output and increases rollout diversity, which is similar to increasing the decoding temperature.”

— the ESRL paper