Glonce

Sparse autoencoders show Qwen3-30B-A3B MoE experts specialize finely

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact40Novelty60Relevance100Surprise35Players20Credibility75
Sparse autoencoders show Qwen3-30B-A3B MoE experts specialize finely

Mixture-of-Experts (MoE) models send each token to a few of many experts, but what any single expert is responsible for is still unsettled. The paper argues that existing analyses count how often an expert is selected on corpora labeled by domain. In the authors' words, that approach fixes the taxonomy in advance, stays coarse, and records selection rather than contribution.

The authors study specialization in feature space instead. They train a sparse autoencoder (SAE) on the output of an MoE layer. That output is a router-weighted sum of the experts' outputs, and the SAE encoder is affine, so the pre-activation of every SAE latent decomposes exactly over the experts. Aggregated over a corpus, this gives each expert a dominant set of latents whose activation it principally drives.

The analysis was run on layers 4, 30 and 46 of Qwen3-30B-A3B. There, the authors report that contribution is far more concentrated than gating, that most latents have only one or two dominating experts, and that the overlap between dominant sets concentrates on a few expert pairs.

Auto-interpreting these dominant sets reveals what the authors call a division of labor below coarse domains. Examples include distinct experts for Java and for front-end web code, and distinct experts for geometry and for arithmetic and polynomials. The coarse domains themselves take very unequal numbers of experts. The SAEs and code are available in the GitHub repository Bytedance-Seed/SAE-Interprets-MoE.

Key facts

  • A sparse autoencoder trained on an MoE layer's output lets each latent's pre-activation decompose exactly over the experts, because the output is a router-weighted sum and the encoder is affine.
  • On layers 4, 30 and 46 of Qwen3-30B-A3B, contribution is far more concentrated than gating, and most latents have only one or two dominating experts.
  • Auto-interpreted expert sets show finer division of labor than coarse domains: separate experts for Java and front-end web code, and for geometry versus arithmetic and polynomials.
  • Coarse domains take very unequal numbers of experts, and overlap between dominant sets concentrates on a few expert pairs.
  • SAEs and code are public at github.com/Bytedance-Seed/SAE-Interprets-MoE.

Why it matters

What a single MoE expert actually does has been hard to pin down. Earlier analyses, as the authors describe them, count how often an expert is selected on domain-labeled corpora. That fixes the categories in advance, stays coarse, and measures selection rather than contribution. This work measures what each expert contributes in feature space, and finds structure finer than domain labels such as code or math.

Who it affects

Interpretability researchers and anyone studying how MoE models divide work among experts. The analysed model is Qwen3-30B-A3B, so the findings bear most directly on that model's layers 4, 30 and 46.

How to use it

The authors release their SAEs and code at https://github.com/Bytedance-Seed/SAE-Interprets-MoE. Researchers can use them to repeat the expert-level decomposition or apply the method to MoE layers of their own.

How solid is it

This is a paper page on Hugging Face and the findings come from the authors' abstract. The claims are specific: contribution is more concentrated than gating, and most latents have one or two dominators. No numeric concentration metrics, expert counts per domain or SAE sizes are given in the source text. The method for the auto-interpretation is not described there either.

Risks and caveats

Only layers 4, 30 and 46 of one model, Qwen3-30B-A3B, are mentioned, so how far the pattern holds elsewhere is not shown in the source. No claim is made that the findings improve model performance, routing or training. The Java and front-end web code examples, and the geometry versus arithmetic and polynomials examples, are illustrations of the authors' interpretation of the dominant sets.