smolmaxxing patch runs 8B and 16B MoE models on about 1 GB of RAM

smolmaxxing patch runs 8B and 16B MoE models on about 1 GB of RAM

smolmaxxing is a repository that patches llama.cpp to run the biggest language model possible on the smallest hardware you own. Its README starts from a simple observation: TV boxes, old tablets, single-board computers and routers all have a CPU, some RAM and flash storage that sit idle most of the day. The tool finds the biggest model each device can really run, then pushes it further with custom kernels, careful memory use and weights streamed from flash. The authors say every number in the repo was measured on the device itself.

The headline result: both devices the README covers run 8B and 16B mixture-of-experts models on about 1 GB of RAM. They do it by keeping only part of each model in memory and reading the rest from flash as needed. The setup guides point to a TV box (or another ARMv8.2 Android device) and a jailbroken iPad on iOS 12.

The README lists four main ideas. First, kernels built for the chip: a 1-bit lookup-table kernel, repacked 2-bit and ternary formats for the Cortex-A55, and versions of them for the A7, which has no dot-product instructions. Second, less RAM per model: most of a small model's vocabulary is rarely used, so the project drops those output rows, and on the TV box a "headless" mode also stops the launcher and Google services while a model runs. Third, experts paged from flash: a mixture-of-experts model uses only a few experts per token, so the patch keeps the popular experts in RAM, reads the others from flash on demand and prefetches the next layer's experts. The README says this is how 8B and 16B models run on 1 GB. Fourth, streaming for dense models bigger than RAM, at the flash's full speed.

The authors say the techniques are not tied to one box: the 1-bit kernel needs only basic NEON, and flash streaming works on any Linux or Android device with more model than RAM. More devices, including phones, single-board computers and routers, are listed as coming. A docs/how-it-works.md file describes each technique with its measured gain.

The quickest start is a ready-made model file from Hugging Face, but these files need the patched llama.cpp; stock llama.cpp cannot load them. Setup guides exist for a TV box or other ARMv8.2 Android device (docs/tv-box-setup.md) and for the jailbroken iPad (ipad-air/README.md), and docs/models.md lists all model files and how to rebuild them. The code is MIT licensed, the llama.cpp patches are derivatives of llama.cpp (MIT), model weights keep their own licenses, and the evaluation texts are Wikipedia excerpts (CC BY-SA 4.0). The project builds on llama.cpp and llama2.c and credits models from Qwen, unsloth, PrismML (Bonsai), inclusionAI (Ling-mini-2.0), tablinum (Moon-Beam), bartowski, mradermacher, Google (Gemma) and stories260K.

Key facts

  • smolmaxxing is a patch on llama.cpp that aims to run the biggest possible language model on small, old hardware: TV boxes, old tablets, single-board computers and routers.
  • The README says both covered devices run 8B and 16B mixture-of-experts models on about 1 GB of RAM, keeping part of each model in memory and reading the rest from flash.
  • Popular experts stay in RAM, others are read from flash on demand, and the next layer's experts are prefetched; a mixture-of-experts model uses only a few experts per token.
  • Other techniques: a 1-bit lookup-table kernel, repacked 2-bit and ternary formats for the Cortex-A55 (with A7 versions), dropped rarely used vocabulary rows, and a headless mode on the TV box.
  • Ready-made model files need the patched llama.cpp, since stock llama.cpp cannot load them; the project is MIT licensed and model weights keep their own licenses.

Why it matters

Many people own idle hardware with a CPU, some RAM and flash storage. This project argues that such devices can run far larger models than their RAM suggests, if the model is paged in from flash as it is used. Running 8B and 16B mixture-of-experts models on about 1 GB of RAM is the concrete claim. The approach leans on how mixture-of-experts models work: only a few experts are used per token, so most of the weights can wait in flash.

Who it affects

Hobbyists and developers with old Android TV boxes, tablets, or jailbroken iPads on iOS 12, and anyone who wants local models on low-memory ARM devices. The README lists phones, single-board computers and routers as devices still to come. It says the 1-bit kernel needs only basic NEON and flash streaming works on any Linux or Android device with more model than RAM.

How to use it

The quickest start is a ready-made model file from Hugging Face, run with the patched llama.cpp from the repo, since stock llama.cpp cannot load these files. For a TV box or another ARMv8.2 Android device, follow docs/tv-box-setup.md. For a jailbroken iPad on iOS 12, follow ipad-air/README.md. docs/models.md lists all model files and how to rebuild them, and docs/how-it-works.md covers every technique with its measured gain. The code is MIT licensed; model weights keep their own licenses.

How solid is it

The authors state that every number in the repo was measured on the device itself, and the repository includes documentation of the techniques and their gains. The 1 GB figure is given as approximate. The visible text gives no tokens-per-second or other speed figures, and no accuracy or perplexity figures. It does not say which model runs on which device, name the exact 16B model, or compare against stock llama.cpp or other runtimes.

Risks and caveats

The model files only work with the patched llama.cpp, so they are tied to this fork. The visible text does not say how much slower flash-streamed inference is than in-RAM inference, so the practical usability of 8B and 16B models on 1 GB is unclear from the README text alone. The 8B and 16B results are for mixture-of-experts models; dense models bigger than RAM are handled by a separate streaming path. Model weights keep their own licenses, so check each one before use.

“Run the biggest language model you can on the smallest hardware you own.”

— smolmaxxing README