Glonce

Mixture of Wide Experts (MoWE) builds readable coordinates into language models

researchThe Glonce editorsarXiv🔥58Glonce rating58Impact40Novelty60Relevance100Surprise30Players10Credibility75

Language models often spread one concept across many internal dimensions, which makes their features hard to interpret. The usual remedy is a sparse autoencoder (SAE), which recovers clearer features after training. The authors of this arXiv paper ask a different question: can a model learn interpretable output coordinates directly?

Their answer is Mixture of Wide Experts (MoWE), a sparse feed-forward layer in which the selected experts read the full representation but write to fixed, disjoint output coordinates. On a controlled task with known hidden causes, the authors say the design preserves predictive performance while localizing causes more clearly than dense and standard mixture-of-experts (MoE) models.

For language modelling, they train two 19-layer runs of each architecture for 10 billion tokens at approximately matched compute. MoWE reaches an evaluation cross-entropy of 2.417 against 2.452 for the dense model (lower is better). Standard MoE reaches 2.22, which is lower than both.

Readability is measured with a language-model judge. The judge describes each coordinate from passages where it is strongly active, and that description is then tested on new passages. After correcting for descriptions that also fit other coordinates, the specificity score is 0.523 for MoWE, 0.375 for dense and 0.268 for standard MoE. TopK SAEs trained on the same feed-forward updates score 0.489 for the dense model and 0.556 for MoWE. The authors conclude that native MoWE coordinates are comparable to the dense model with an SAE.

Two case studies show the coordinates being used. In the first, the authors identify a four-coordinate causal path that supports the model's prediction of "degrees" after the prompt "A right angle is 90". They then copy seven angle-related coordinate values into the model while it processes "A rectangle has four", and its next-token prediction changes from "sides" to "degrees". In the second, one fixed coordinate set detects hazardous requests and another steers the next token toward refusal. The authors say these examples show that MoWE combines sparse conditional computation with coordinates that can be identified and reused directly.

Key facts

  • MoWE is a sparse feed-forward layer: selected experts read the full representation but write to fixed, disjoint output coordinates.
  • In 19-layer runs on 10 billion tokens at approximately matched compute, MoWE reaches evaluation cross-entropy 2.417 versus 2.452 for dense; standard MoE reaches 2.22.
  • LM-judge specificity of native coordinates: 0.523 for MoWE, 0.375 for dense, 0.268 for standard MoE; TopK SAEs score 0.489 on dense and 0.556 on MoWE.
  • Copying seven angle-related coordinate values into the model changed its next-token prediction on "A rectangle has four" from "sides" to "degrees".
  • A second case study finds one fixed coordinate set that detects hazardous requests and another that steers the next token toward refusal.

Why it matters

Interpretability work today mostly happens after training: a sparse autoencoder is fitted to a model's activations to pull out cleaner features. MoWE tries to move that step into the architecture, so that the model's own output coordinates are already readable. If it holds up, a feature could be located and edited directly in the model rather than through a separate learned decomposition. The paper also keeps the sparse, conditional computation that makes MoE layers attractive.

Who it affects

Mainly interpretability researchers and people designing sparse model architectures. The refusal case study, where one coordinate set detects hazardous requests and another steers the next token toward refusal, points at safety work as a possible user of such coordinates, though the paper reports it only as an example.

How to use it

The abstract describes a method rather than a product. The idea to take away is the layer design: experts that see the whole representation but each own a fixed, separate slice of the output. A code or weights release is not mentioned in the source, so reproducing the results would mean building the layer from the paper's description.

How solid is it

The numbers come from the authors' own abstract, from two 19-layer runs per architecture trained for 10 billion tokens. On the readability score, MoWE (0.523) beats the dense model (0.375) and standard MoE (0.268), and it sits near the dense model with an SAE (0.489). Readability is judged by a language model, not by humans. The size of the controlled task with known hidden causes is not described, and the hazardous-request case study gives no quantitative results.

Risks and caveats

Standard MoE reaches a lower cross-entropy (2.22) than MoWE (2.417) and dense (2.452), so MoWE does not match the best predictive result in the comparison; its edge over dense is 0.035. "Approximately matched compute" is not quantified. Model parameter counts, expert counts and the number of active experts are not given. The abstract does not name the judge model or the dataset. The two case studies are single examples, so they show that the coordinates can be used, not how often that works.

“We ask whether models can instead learn interpretable output coordinates directly.”

— From the abstract of the MoWE paper on arXiv