Meta Expert paper shares MoE experts across layers, matching baseline with 70 to 80% of them

A paper on Hugging Face proposes a change to how Mixture-of-Experts (MoE) models are built. MoE approaches activate only part of a model for each input and have become essential for scaling model parameters. But mainstream models use hierarchical architectures, where each layer has its own set of experts. The authors say these experts, although topologically independent, show substantial similarity and redundancy in what they store.

Their answer is the Meta Expert: an expert shared between layers, in contrast to the hierarchically arranged Local Expert used in existing mainstream models. The authors hypothesize that inter-layer shared Meta Experts hold greater potential than Local Experts.

To test this, they ran a multi-dimensional empirical comparison between Local and Meta Experts. They report that models with Meta Experts show lower parameter redundancy, lower cross-entropy loss and higher downstream task accuracy. The headline result: with the activated quantity and the expert granularity held fixed, their model matches the performance of Vanilla MoE using only 70 to 80% of the experts. The authors say this substantially improves the model's scaling potential.

The paper also offers a theoretical argument. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, the authors' theoretical analysis shows that Meta Experts give hidden states more flexible residual pathways than Local Experts do, which they say improves information retrieval efficiency.

The text available is the abstract only. It gives no model sizes, datasets, benchmark names, or exact accuracy and loss values; those are reported only as lower or higher.

Key facts

The paper proposes Meta Experts: MoE experts shared across layers, as opposed to the layer-local experts in mainstream hierarchical MoE models.
The authors say experts in different layers of current MoE models, though topologically independent, show substantial similarity and redundancy in stored content.
With activated quantity and expert granularity held fixed, the authors report matching Vanilla MoE using only 70 to 80% of the experts (a share of the expert count, not an accuracy figure).
In their empirical comparison, Meta Experts bring lower parameter redundancy, lower cross-entropy loss and higher downstream task accuracy.
The authors' theoretical analysis says Meta Experts give hidden states more flexible residual pathways, improving information retrieval efficiency.

Why it matters

MoE is a standard route to scaling models, and the paper targets a cost inside it: experts in different layers storing similar content. If experts can be shared across layers, the same quality may need fewer of them. The authors frame the 70 to 80% figure as a substantial gain in scaling potential. The idea is to remove inter-layer redundancy among experts and improve information storage efficiency, while also improving how hidden states retrieve information.

Who it affects

The direct audience is researchers and engineers who design or train sparsely activated MoE models, since the proposal changes how experts are arranged across layers. The source names no authors, institutions or products, so nothing here ties the work to a particular lab or release.

How to use it

This is an architecture proposal, not a tool. The source mentions no code, model release or training-cost figures. A practitioner could treat the Meta Expert versus Local Expert comparison as a design idea to check against their own MoE setup, keeping activated quantity and expert granularity fixed as the authors did.

How solid is it

The claims come from the authors' own abstract: an empirical comparison plus a theoretical analysis. The text gives no model sizes, datasets or benchmark names, and no specific accuracy or cross-entropy values, only that they are lower or higher. It does not say which Vanilla MoE baseline or how many experts were used. The 70 to 80% figure is a share of expert count, not an accuracy score.

Risks and caveats

The headline result rests on a fixed activated quantity and expert granularity; the source does not say whether it holds under other settings. No inference-speed or memory measurements are reported; only expert count and storage and retrieval efficiency are discussed. The claim that Meta Experts hold greater potential is stated by the authors as a hypothesis they then test, and independent replication is not mentioned in the source.

“experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content”

— Meta Expert paper abstract