Glonce

Edge0 runs 35B-parameter MoE models from SSD on a single 24GB GPU

researchThe Glonce editorsHugging Face🔥74Glonce rating74Impact65Novelty78Relevance100Surprise52Players30Credibility82
Edge0 runs 35B-parameter MoE models from SSD on a single 24GB GPU

Researchers have presented Edge0, a streaming inference engine for mixture-of-experts (MoE) models that lets a 35B-parameter model run on a single 24GB machine by keeping its weights on SSD instead of loading them all into memory.

The problem Edge0 targets is that MoE inference on consumer hardware is bounded by weight memory rather than compute. A 35B-class MoE model takes 19.5GB at 4-bit quantization, and sparsity shrinks the compute needed per token, not the number of bytes that must be held. Simply offloading unused expert weights to SSD does not fix this on its own, the authors say, because the experts needed for layer N+1 cannot be chosen until layer N has already produced its output, so the SSD reads cannot start early enough to be hidden behind computation.

Edge0's answer is a prerouter: a small per-layer head that predicts, one token ahead, which experts the next layer will route to. That prediction is not just a hint for prefetching, it is consumed as the actual routing decision, so the set of experts staged into memory always equals the set actually used and none of the staged work is discarded.

Because the model runs at 4-bit precision and its routing is replaced by a prediction rather than computed exactly, this can cost output quality. Edge0 recovers it with an unmerged recovery LoRA trained on the student model's own path, which the authors say pays back the quality lost to int4 quantization and to the routing substitution.

On a single 24GB machine, Edge0 serves the 35B MoE model at 20 tokens per second while staying within 3GiB of peak active memory, with output the authors report as within a few points of the same model's full fp16 version on average across five public benchmarks. The same framework also runs an 8B-parameter model. Edge0's framework, model checkpoints and adapters have been released as open source.

Key facts

  • Edge0 serves a 35B-parameter MoE model from SSD on a single 24GB machine at 20 tokens per second inside 3GiB of peak active memory.
  • The bottleneck it targets: a 35B-class MoE model needs 19.5GB at 4-bit quantization, and sparsity cuts compute per token but not the memory footprint.
  • A per-layer prerouter predicts the next layer's expert routing one token ahead, and that prediction is used as the actual routing, so the staged and routed expert sets always match.
  • An unmerged recovery LoRA trained on the student path recovers quality lost to int4 quantization and to the routing substitution, landing within a few points of the fp16 teacher across five public benchmarks.
  • An 8B-parameter tier also runs on the same framework, and Edge0's framework, checkpoints and adapters are open source.

Why it matters

MoE inference on consumer hardware is currently limited by how much weight memory a machine has, not by how much compute it can do, since sparsity only reduces the compute per token. Edge0 attacks that memory bound directly: by predicting routing ahead of time instead of computing it just in time, it makes it possible to keep most of a 35B-parameter model's weights on SSD and still serve it in real time, rather than requiring enough RAM or VRAM to hold the whole model.

Who it affects

The main audience is people trying to run large MoE models on consumer-grade hardware rather than server GPUs, such as individual developers, researchers or small teams working on a single 24GB machine. It also matters to anyone building or maintaining local inference tooling for open-weight MoE models, since Edge0's approach is offered as a released framework rather than a one-off demo.

How to use it

Edge0's framework, checkpoints and adapters are released as open source, and the same setup also runs an 8B-parameter model tier. The source does not give a price, license terms or specific GPU model beyond describing the test machine as having 24GB of memory.

How solid is it

The result comes from the paper's own authors, whose names and institutional affiliation are not given in the available text. The reported quality, within a few points of the fp16 teacher on average across five public benchmarks, and the throughput and memory figures are the authors' own measurements; the five benchmarks are not individually named, and no comparison against a named competing offloading system or baseline numbers is given.

Risks and caveats

The quality claim is an average approximation, "within a few points", rather than an exact parity figure, so results could vary by benchmark or task. Without named benchmarks, a named baseline system, or author and institutional attribution, the result is harder to independently verify or reproduce until more detail becomes available.

“Naive offloading to SSD does not help on its own, because layer N+1's experts must be chosen before layer N's output exists, so the reads cannot start early enough to hide behind compute.”

— the paper's authors