Glonce

Slotstream runs 104GB Qwen3.8-Flash-Next model on 48GB Macs

toolsThe Glonce editorsHacker News🔥70Glonce rating70Impact50Novelty65Relevance92Surprise62Players35Credibility87
Slotstream runs 104GB Qwen3.8-Flash-Next model on 48GB Macs

A developer who goes by carloslfu published slotstream, a single-binary Swift tool with no Python dependency, for running Qwen3.8-Flash-Next, a 125B-parameter mixture-of-experts model that takes up 104GB on disk at 4-bit quantization (103.8GB across 24 files), on Apple Silicon Macs that do not have enough memory to hold the whole model at once. The trick is streaming: slotstream reads experts from SSD with pread into a fixed pool of cache slots shared across the model's 48 layers, so hot layers borrow slots from cold ones and the process never needs more memory than the cache it is given. The dense trunk of the model, 3.8GB, stays resident; the bulk of the weights, 68GB of routed experts (512 per layer, 10 active per token) plus a 32GB n-gram table, gets streamed on demand.

The tool speaks the Ollama and OpenAI chat APIs on port 11434, so Open WebUI and OpenAI SDK clients work against it unchanged; the Ollama CLI itself cannot connect in the current 0.2.0 release because its requests carry fields the strict validator rejects, though the fix is already on the main branch and will ship in the next release. Unsupported features, tools, images, JSON-schema output and logprobs, return a clear 400 rather than being silently ignored.

With no flags, slotstream auto-sizes itself. On the reference 48GB Mac it targets 33GB total process memory and caches about 152 of the model's 512 experts per layer. Auto-sizing takes the lowest of three limits, a 33GB cap, 70% of RAM, or 2GB under the Metal working-set limit, and the developer found in a gigabyte-at-a-time sweep that nothing between 34GB and 84GB of target memory decoded or prefilled any faster, so a 128GB Mac gets the same plan as a 48GB one. Requirements are Apple Silicon, macOS 14 or later and roughly 110GB of free disk; because disk, not RAM, is the binding constraint, the developer calls a 512GB Mac the realistic minimum. Only the 48GB tier of the memory and performance table comes from a real run, on an M5 Pro; every smaller tier is an estimate extrapolated from that machine's curve, and smaller Macs also carry slower SSDs, a further reason those numbers could be optimistic.

Decode speed on the measured 48GB machine is about 12 tokens per second warm. Prompt processing is the slower stage: all of it runs before the first token appears, so an 8,000-token prompt waits about a minute on a 48GB Mac and over three minutes on a 16GB one, though follow-up turns in the same conversation only prefill what changed, cutting the eight-turn time to first token from 25.8 seconds down to 6.0 seconds at a 16GB target. Prompt plus completion is capped at 32,768 tokens. An optional speculative-decoding mode, MTP, drafts several tokens ahead with a bundled head and verifies them in one pass; its first guess lands 86% of the time, which the developer estimates as a 1.5x to 1.9x speedup once the target exceeds about 26GB, though that projection is arithmetic from the accept rate rather than a measured A/B result. Below that threshold the measured effect is a 0.96x slowdown, so auto-sizing disables MTP under roughly 26GB of target memory; enabling it costs 1.6GB of extra cache and requires a one-time conversion that pulls 4.9GB from the official release.

The developer explains the approach was necessary because Apple's MLX framework cannot partially load a memory-mapped tensor: since a top-10 expert gather has to evaluate all 512 experts in a layer, a plain mmap path ends up loading roughly 100GB and crashes, and the stock mlx_lm.load() route pushed the same 48GB test machine into 48GB of swap without producing a single token. Cache size in slotstream changes only speed, not output: greedy decoding is byte-identical between a 4GB cache and a 24GB one, a claim backed by a standing test. Reused prefix-cache state is not bit-identical to recomputing from scratch, so replies can occasionally differ where two candidate tokens were nearly tied; a --no-prefix-cache flag disables reuse for exact reproducibility.

Installation is a curl-to-shell script that installs to ~/.slotstream/bin, and the weights download separately, over eight parallel TCP connections, at 112MB/s and 16 minutes measured on a 1Gbit/s datacenter link, with estimates of about 2 hours 20 minutes at 100Mbps and about 9 hours at 25Mbps. Interrupted downloads resume from where they stopped, and all 24 weight files are checked against sha256 hashes compiled into the binary. The v0 release runs exactly one model, qwen3.8-flash-next:4bit, with a per-user lock allowing only one model process at a time; macOS 14 and 15 have had only the installer exercised, not the runtime itself. The code is MIT licensed; the weights, mirrored from pipenetwork/Qwen3.8-Flash-Next-MLX-4bit, remain under the Qwen community license.

Key facts

  • Slotstream streams the 104GB, 125B-parameter Qwen3.8-Flash-Next mixture-of-experts model (103.8GB across 24 files) from SSD into a fixed cache pool, letting it run on Macs with as little as 48GB of RAM instead of needing the whole model in memory.
  • On the sole machine it has been measured on, an M5 Pro with 48GB RAM, it decodes at about 12 tokens per second warm; every smaller tier in its published table is an extrapolation, not a real run.
  • Auto-sizing caps the memory target at the lowest of three limits (33GB, 70% of RAM, or 2GB under the Metal working-set limit), because a gigabyte-at-a-time sweep found nothing between 34GB and 84GB decoded or prefilled any faster.
  • An optional speculative-decoding mode drafts tokens ahead with a bundled head that guesses right 86% of the time, projected at 1.5x to 1.9x faster above a 26GB memory target, though that figure is arithmetic from the accept rate rather than a measured benchmark.
  • The full weight download runs over eight parallel connections, measured at 112MB/s and 16 minutes on a 1Gbit/s datacenter link, and every downloaded file is checked against sha256 hashes compiled into the binary.

Why it matters

Apple's MLX framework cannot partially load a memory-mapped tensor, so a plain mmap approach to a mixture-of-experts model like Qwen3.8-Flash-Next ends up loading close to the full 100GB and crashing; the developer reports that the stock mlx_lm.load() route pushed the same 48GB test Mac into 48GB of swap without producing a single token. Slotstream avoids that failure by reading experts from disk on demand into a fixed-size cache pool shared across all 48 layers, so hot layers borrow slots from cold ones and total memory use stays capped regardless of the model's full size. The result is a model roughly twice the size of the Mac's RAM running at a usable, if modest, 12 tokens per second.

Who it affects

Developers and hobbyists who want to run a large local mixture-of-experts model on Apple Silicon without buying a Mac with very large unified memory. It requires Apple Silicon and macOS 14 or later, though only the installer has been exercised on macOS 14 and 15, not the inference runtime itself; disk space is the binding constraint at roughly 110GB free, which the developer says makes a 512GB Mac the realistic minimum regardless of RAM.

How to use it

Install with a curl-to-shell script that places the binary in ~/.slotstream/bin and adds it to PATH; slotstream doctor previews the memory plan for a machine before anything downloads, and pull or the first run fetches the 103.8GB of weights after confirming there is enough disk space. slotstream serve exposes the Ollama and OpenAI chat APIs on port 11434, so Open WebUI and OpenAI SDK clients work unmodified today; the Ollama CLI itself cannot connect in the current 0.2.0 release, though the underlying validator issue is already fixed on the main branch and due to ship in the next release. Unsupported request features, tools, images, JSON-schema output and logprobs, return a clear 400 error rather than failing silently. The code is MIT licensed; the model weights, mirrored from pipenetwork's Qwen3.8-Flash-Next-MLX-4bit repository, remain under the separate Qwen community license.

How solid is it

Only the 48GB memory tier is measured on real hardware, an M5 Pro; every smaller tier in the published table is an estimate extrapolated from that one machine's curve, and the developer notes smaller Macs also have slower SSDs, which could make those estimates optimistic. The project ships its own verification battery covering weight provenance, byte-equality of output across different cache sizes and live cache resizes, the speculative-decoding gates, and a serving-robustness suite built from inputs that previously crashed the server, plus a separate end-to-end test of the install script and the resulting binary.

Risks and caveats

The v0 release runs exactly one model, qwen3.8-flash-next:4bit, and a per-user lock permits only one model process at a time. macOS 14 and 15 support currently covers only the installer, not the inference runtime. The Ollama CLI cannot connect until the next release ships the validator fix already merged to main. Reusing cached prefix state across conversation turns is not bit-identical to recomputing it from scratch, so a reply can occasionally differ where two candidate tokens were nearly tied in probability, a behavior that can be turned off with --no-prefix-cache at the cost of speed. Enabling the speculative-decoding mode requires a one-time conversion that pulls an extra 4.9GB from the official release and must be run from a cloned copy of the repository with its Python environment.

“Cache size changes speed, never output.”

— slotstream project documentation