Glonce

Swiftlet runs 80B Qwen on 4.3GB of Mac RAM, 35B on iPhone

toolsThe Glonce editorsHacker News🔥68Glonce rating68Impact68Novelty77Relevance98Surprise72Players40Credibility82
Swiftlet runs 80B Qwen on 4.3GB of Mac RAM, 35B on iPhone

A Show HN post introduces Swiftlet, a Swift and Metal runtime built for the Qwen3-Next and Qwen3.5/3.6 Mixture-of-Experts (MoE) hybrid model family, which the developers say runs the 80B parameter model on a Mac using about 4.3 GB of RAM even though the model's container is 42 GB on disk, and runs the 35B parameter model on an iPhone 17 in about 2.5 GB of RAM. On the iPhone, the 35B model currently generates at about 1 token per second; the developers write that as far as they know, this is the first time a model of this class has run natively on a phone.

The trick is that Swiftlet keeps only the small dense core of a model resident in memory, roughly 1.3 GB for the 35B model and 2.5 GB for the 80B model at 4-bit precision, and streams the routed MoE expert weights from SSD storage on demand. Only about 3B parameters are active per token: each layer routes a token to 10 of 512 experts in the 80B model, or 8 of 256 experts in the 35B model. The developers note this tradeoff plainly: these models chat and write like large models but recall facts like small ones, since so few parameters are active at once. Experts are repacked into a custom .qpack container with fixed-stride blocks so that fetching one expert takes exactly one read from SSD, with no memory-mapping and no page-cache thrashing, and a bounded cache pool evicts by least-frequently-used plus recency; the developers measured 43 to 70 percent cache hit rates at the same throughput, and say cache size barely affects speed because Apple SSDs absorb the misses. About 75 percent of the model's layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so those layers carry no growing key-value cache at any context length. The forward pass runs entirely on Metal with runtime-compiled shaders, so no Metal toolchain is required at build time and the same code runs on both macOS and iOS.

Requirements are Apple Silicon with macOS 14+ or iOS 17+, plus free SSD space for the model container: 18 GB for the 35B model and 42 GB for the 80B model. The project ships as a Swift package (SwiftletCore) for developers to embed in their own apps, a command-line tool for chat and one-shot generation, a local OpenAI-compatible server on loopback, and an iOS app called Priv AI that lets users download and chat with the 35B model on-device with no server involved; that app's Experimental Models feature was still in App Store review at the time of posting, so the developers also point to an open source app, leonickson1/localLLM, that can be built from source today. The developers say Swiftlet's roughly 10,000 lines of Swift and Metal code were written from scratch against mlx-lm reference implementations, adapting design lessons on expert streaming, caching and container packing that the TurboFieldfare project had earlier published for Gemma models, while building support for the different Qwen hybrid architecture, its own quantization kernels, container format, chat session layer and iPhone integration. Swiftlet was built in collaboration with Claude Code and is released under the Apache 2.0 license; the Qwen model weights it downloads separately are also Apache 2.0.

Key facts

  • Swiftlet runs the 80B parameter Qwen3-Next model on a Mac in about 4.3 GB of RAM, despite a 42 GB on-disk container, by streaming routed Mixture-of-Experts weights from SSD instead of loading them all into memory.
  • It runs the 35B parameter Qwen3.5/3.6 model on an iPhone 17 in about 2.5 GB of RAM at about 1 token per second, which the developers say is, as far as they know, the first time a model of this class has run natively on a phone.
  • Only about 3B parameters activate per token (10 of 512 experts per layer for the 80B model, 8 of 256 for the 35B model), and the hot-expert cache measured 43 to 70 percent hit rates at the same throughput.
  • Requirements are Apple Silicon with macOS 14+ or iOS 17+ and 18 GB (35B) or 42 GB (80B) of free SSD space; the code is Apache 2.0, built in collaboration with Claude Code, and iPhone users can try it through the open source Priv AI app.
  • About 75 percent of the model's layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so those layers carry no growing key-value cache at any context length.

Why it matters

Running MoE models of this size on ordinary consumer hardware, rather than a server GPU, is the notable part: Swiftlet's approach of keeping only a small dense core in RAM and pulling routed expert weights from SSD on demand lets an 80B parameter model fit in about 4.3 GB of memory, and lets a 35B model run natively on a phone at all. That reframes the constraint on running large MoE models locally from total RAM to SSD read speed and layout, which is a different and, per the cache hit-rate numbers here, more forgiving bottleneck on Apple hardware.

Who it affects

iOS and macOS developers building on-device AI features can embed the SwiftletCore Swift package directly, or point an existing OpenAI-compatible chat client at the local loopback server. End users without development tools can try the 35B model through the Priv AI iPhone app once its Experimental Models feature clears App Store review, or build the open source companion app from source today. It is not aimed at general consumers yet: at about 1 token per second on iPhone, the phone experience is a proof of concept rather than a practical chat speed.

How to use it

The code is on GitHub under an Apache 2.0 license. Building it requires Apple Silicon on macOS 14+ or iOS 17+; a swift build -c release produces the CLI, and a repack tool downloads and converts model checkpoints from Hugging Face (or local MLX checkpoints) into Swiftlet's own .qpack container format, resumable if interrupted. Free SSD space of 18 GB is needed for the 35B model or 42 GB for the 80B model. The CLI supports interactive chat and one-shot generation with stats, and swiftlet-server exposes an OpenAI-compatible chat-completions endpoint on loopback for any compatible chat UI. On iPhone, the same 35B model is reachable through the Priv AI app or by building the open source localLLM app from source.

How solid is it

This is a Show HN post from the project's own developers rather than an independent write-up, and the numbers (RAM use, cache hit rates, the 1 token per second figure) are the developers' own measurements with no outside benchmark cited. That said, the post describes a real correctness process: every layer of the forward pass, including the Gated DeltaNet recurrence, gated attention and sparse MoE routing, is checked against mlx-lm reference implementations with per-layer fixtures in both full precision and int4 quantized form; incremental decoding is verified against whole-sequence processing; Metal kernels are tested against a CPU reference; and the model containers are described as byte-verifiable against their source checkpoints. The claim of being the first model of this class to run natively on a phone is explicitly hedged by the developers themselves ("as far as we know") rather than presented as a verified record. At the time of the post the discussion had drawn 48 points and 18 comments on Hacker News.

Risks and caveats

Because only about 3B parameters are active per token regardless of the model's total size, the developers themselves flag that these models chat and write fluently but recall facts more like a small model than a large one. The iPhone decode speed, about 1 token per second for the 35B model, is slow enough to limit real use today, and the developers describe the current bottleneck as dispatch bound rather than IO bound, meaning the SSD streaming trick is not yet the limiting factor. The phone app's Experimental Models feature was still in App Store review at the time of the post, so it may not be available to iPhone users for a few days. The project runs only on Apple Silicon hardware and offers no comparison against other local-inference runtimes, so how it stacks up outside the Apple ecosystem, or against alternatives on the same hardware, is not addressed.

“As far as we know, that is the first time a model of this class has run natively on a phone.”

— Swiftlet's developers, in the Show HN post