Glonce

LatentLoop looped MoE predicts 19.0% compute savings over vanilla MoE

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact45Novelty60Relevance100Surprise30Players10Credibility75
LatentLoop looped MoE predicts 19.0% compute savings over vanilla MoE

Looped Mixture-of-Experts (MoE) models reuse the same parameters across several passes, or visits, while keeping computation sparse, and this can improve performance under matched budgets. The paper starts from the observation that different loop visits can play different roles even when they share parameters.

To probe those roles, the authors train a model that applies a shared block twice and learn a separate residual coefficient for each visit. The first visit ends up with a smaller learned scale. Fixed-coefficient experiments then show that reducing the first visit's scale preserves or improves performance, while the same reductions applied to the second visit hurt it. That asymmetry motivates the proposal: allocate compute unevenly between visits.

The resulting design, LatentLoop, compresses the sequence in the first visit, retains full resolution in the second, and reallocates the saved computation to increase the hidden size. Two further pieces complete it. The model learns how strongly the first visit updates the full sequence, and it shares MoE weights across visits while using separate attention parameters, so that the two visits can work at different sequence resolutions.

The evaluation covers 15 configurations spanning 1.3B to 7B parameters, with the largest budget exceeding 10^21 FLOPs. Across them, LatentLoop consistently lowers validation NLL. At a fitted NLL of 1.988, the predicted compute savings are 19.0% relative to vanilla MoE and 7.9% relative to the looped baseline. LatentLoop also improves average scores on nine benchmarks at all sizes, in both 0-shot and 5-shot settings. The authors conclude that the results suggest asymmetric compute allocation is a promising direction for looped MoE models.

Key facts

  • LatentLoop compresses the first loop visit's sequence, keeps full resolution in the second, and spends the saved compute on a larger hidden size.
  • The idea comes from a probe: the first visit learns a smaller residual scale, and reducing it preserves or improves performance while the same cut to the second visit hurts.
  • Tested across 15 configurations from 1.3B to 7B parameters, with the largest budget above 10^21 FLOPs, it consistently lowers validation NLL.
  • At a fitted NLL of 1.988, predicted compute savings are 19.0% versus vanilla MoE and 7.9% versus the looped baseline.
  • Average scores improve on nine benchmarks at all sizes in 0-shot and 5-shot settings.

Why it matters

Looped MoE models try to get more out of a fixed parameter and compute budget by reusing parameters and keeping computation sparse. This paper argues that the loop visits are not equal: the first visit contributes less per unit of compute than the second. Treating them differently, by compressing the first and widening the hidden size with the savings, is presented as a way to push the same budget further. The reported predicted savings of 19.0% over vanilla MoE and 7.9% over a looped baseline are the headline evidence.

Who it affects

Mainly researchers working on efficient language-model architectures, especially mixture-of-experts and parameter-sharing designs. The tested range of 1.3B to 7B parameters places it in the territory of pretraining studies rather than frontier-scale systems.

How to use it

This is a preprint describing an architecture, not a product. The main ingredients a practitioner would need to reproduce are in the abstract: compress the first visit's sequence, keep the second at full resolution, raise the hidden size with the saved compute, learn how strongly the first visit updates the full sequence, and share MoE weights across visits with separate attention parameters. No code, weights or release is mentioned.

How solid is it

The evidence comes from 15 configurations of 1.3B to 7B parameters, with the largest budget exceeding 10^21 FLOPs, and the claim of consistently lower validation NLL is the authors' own. The compute savings are predicted from a fitted curve at NLL 1.988, not measured wall-clock or hardware savings. No absolute NLL values per model or size, and no per-size results, are given beyond the single fitted NLL of 1.988. The nine benchmarks are not named and the size of the average-score improvement is not given. No authors or institutions are named in the source text.

Risks and caveats

The authors themselves word the conclusion cautiously: the results suggest asymmetric compute allocation is a promising direction. The savings figures depend on a fitted curve rather than direct measurement. No comparison to non-MoE dense models or to other looped architectures beyond vanilla MoE and the looped baseline is stated, so how LatentLoop stands against the wider field is not established here.

“These results suggest asymmetric compute allocation is a promising direction for looped MoE models.”

— LatentLoop paper abstract