llama-modchip claims 284B DeepSeek-V4-Flash fits in 64GB RAM

llama-modchip is a GitHub repository (account synchopate) with ggml/llama.cpp kernel patches, quantization transcoders and runtime tools. Its aim is to run sparse Mixture-of-Experts (MoE) checkpoints entirely in system RAM on x86_64 (AVX-512 / AVX2) and ARM64 (NEON / SVE / Apple Metal) processors. The headline claim concerns DeepSeek-V4-Flash, which has 284.33B parameters in total and 13.17B active per token, across 43 MLA layers and 256 routed experts.

The problem it targets: the standard Unsloth UD-IQ1_S build of that model is 76.87 GiB (82.5 GB), more than the roughly 62.5 GiB of usable RAM on a 64 GB Linux machine, so it causes page-cache thrashing or OOM kills. llama-modchip transcodes it into a 56.56 GiB (60.73 GB) checkpoint, 1.71 bits per weight. The README says the whole model plus a 256K-token KV cache then locks into RAM with zero swap, at 57.61-58.9 GiB VmRSS.

How the size drops: the 256 routed experts (the ffn_*_exps tensors) are mostly quantized to IQ1_S (1.5625 bpw) with what the README calls analytical SwiGLU-H2 second-order scale matching, while sensitive boundary layers stay in IQ2_XXS (2.06 bpw) and IQ1_M (1.75 bpw). That saves 20.31 GiB against UD-IQ1_S. The MLA wq_b and attn_output tensors are transcoded to a new type, Fractal-64 (GGML_TYPE_Q2_0, 3.75 bpw, 64-weight blocks with a BF16 scale). Embeddings, the dense layer 0, the 43 shared experts and some MLA projections stay in Q4_K / Q5_K / F32.

By default (v2.4.0 "Lossless") all 6 routed experts per layer are evaluated, with no expert dropping. The README reports a logit cosine similarity of 1.00000 against the unpatched runtime on the same checkpoint. That is a comparison with the same quantized checkpoint, not with the original full-precision model. Setting LLAMA_MOE_TOP_K=5 or 4 is an optional flag that trades a small perplexity change for faster decoding.

Speed claims, measured on the author's AMD Ryzen 9 9955HX (16-core Zen 5) with no GPU. Prefill rises from 25.0 to 44.36 tok/s (+77%) through a 4-level register / L1D / L2 / L3 multi-column tiling schedule, with a register-stationary AVX-512 MQA8 FlashAttention and non-temporal _MM_HINT_NTA streaming of MoE weights so cold experts do not evict hot KV cache lines from L3. Pure decode in the lossless Top-6 mode is 9.06-9.95 tok/s. The README gives +53% for decode from the kernel patches, without a stated baseline in the visible text. The headline range of 9.1-14.7 tok/s includes self-speculative drafting: a 1.64 MB L2/L3 Zobrist n-gram drafter ("Stockfish-PVS v2") lifts decode to 11.5-14.7 tok/s, paired with a fix for speculative rollback of deepseek4 recurrent/MLA states. The flattened benchmark table lists 5.91 tok/s (169.3 ms) for the unpatched UD-IQ1_S build, whose prefill cell reads "OOM / Swap on 64 GB".

Lossy options are tabulated too. The "Sweet-Spot" mode (Top-5 on layers from index 3 up, L2 quadrature) decodes at 9.68-9.87 tok/s with a KL divergence of 0.0042 nats (+0.42% PPL), cosine similarity 0.99714 and 95.8% / 100% Top-1 / Top-3 match. A Top-5 mode with gate-first SwiGLU reaches 10.15 tok/s at 0.0531 nats. The "Turbo 10+" mode (Top-4) reaches 10.12-10.74 tok/s at 0.0281-0.0694 nats.

The repository also covers other models. NVIDIA Nemotron-3.5-30B-A3B (30.2B total, 3.0B active) in a Fractal-64 variant is 14.35 GiB at 3.75 bpw, with a stated 25.0-60.4 tok/s on CPU only. Qwen3-30B-A3B with dynamic Top-4 is listed at 34.80 tok/s with no GPU. A Windows pack splits MoE layers between GPU and CPU. The README also lists results on Threadripper, Apple Silicon, AWS Graviton3 and mini-PC hardware.

Tooling: the ./modchip script offers chat, a web chat with telemetry, a 30-second benchmark scorecard, a quality-and-tool-calling eval, a quantize command, and a claude command that launches Claude Code CLI against the local 256K engine. Snapshots of the KV cache are stored in /dev/shm and restored in 13.1-32.6 ms for 40-tool Claude Code prompts. Because Fractal-64 uses custom tensor type 40, unpatched llama.cpp, Ollama and LM Studio reject these files with "unknown tensor type 40"; users need ./modchip or the prebuilt llama-server-modchip-linux-x86_64 binary from Hugging Face or GitHub Release v2.4.0.

Key facts

The repo shrinks DeepSeek-V4-Flash (284.33B total, 13.17B active) from the 76.87 GiB UD-IQ1_S build to 56.56 GiB at 1.71 bpw, so it can sit in 64 GB of RAM with a 256K KV cache (57.61-58.9 GiB VmRSS).
Reported speeds on a 16-core Ryzen 9 9955HX with no GPU: prefill 25.0 to 44.36 tok/s (+77%), plain decode 9.06-9.95 tok/s, and 11.5-14.7 tok/s with speculative drafting.
The default Top-6 mode evaluates all routed experts and reports 1.00000 logit cosine similarity against the unpatched runtime on the same checkpoint; Top-5 or Top-4 modes are optional and lossy (for example +0.42% PPL for Sweet-Spot).
Files use a custom tensor type (Q2_0, ID 40), so stock llama.cpp, Ollama and LM Studio reject them; the repo's ./modchip launcher or prebuilt Linux x86_64 binary is required.
All benchmarks are the repository's own, on its own hardware.

Why it matters

Running a 284B-parameter MoE model on a desktop-class machine with no GPU is a notable claim. The trick is that only 13.17B parameters are active per token, so the cost is mostly memory capacity rather than compute. The standard 1-bit-class build overshoots a 64 GB machine by about 14 GiB (76.87 GiB against roughly 62.5 GiB usable). This project cuts the checkpoint by 20.31 GiB in the expert tensors and adds CPU kernel work so decoding reaches around 9 to 15 tok/s, usable for interactive chat.

Who it affects

People who want to run large open MoE models locally on CPU-only workstations or laptops with 64 GB of RAM, especially on AMD Zen 5 and other AVX-512 hardware, where the benchmarks were taken. The repo also lists ARM64 and Apple Silicon, a Windows hybrid GPU plus CPU pack, and smaller models such as Nemotron-3.5-30B-A3B (14.35 GiB) and Qwen3-30B-A3B. It also targets people who want a local backend for Claude Code CLI via the ./modchip claude command.

How to use it

On Linux or macOS, clone https://github.com/synchopate/llama-modchip.git, enter the directory and run ./modchip chat. Other commands include ./modchip web (chat UI at http://127.0.0.1:8787), ./modchip bench (a 30-second speed test), ./modchip eval, ./modchip quantize and ./modchip claude. Alternatively, download the prebuilt llama-server-modchip-linux-x86_64 binary and a Fractal-64 .gguf, then run it with options such as -c 262144 -t 16 --port 8080. A CPU readiness check script runs without downloading a model. Windows users start from windows/AVVIA_MODCHIP.bat. The optional LLAMA_MOE_TOP_K=5 or 4 flag trades a small perplexity change for faster decode.

How solid is it

Every number comes from the repository's own README and hardware; the visible text contains no independent or third-party verification. The quality evidence is relative: the 1.00000 cosine similarity compares the default mode with the unpatched runtime on the same heavily quantized checkpoint. The README does not say how a 1.71 bpw model compares in answer quality with the original full-precision DeepSeek-V4-Flash. The headline 9.1-14.7 tok/s range includes speculative drafting; plain decode of the default mode is 9.06-9.95 tok/s. The fetched text is truncated, so the later benchmark tables (including the Nemotron results) are not fully visible.

Risks and caveats

Memory headroom is thin: 57.61-58.9 GiB of resident memory with a 256K context against about 62.5 GiB usable on a 64 GB Linux machine. The speed figures are from a 16-core Zen 5 chip, and other hardware may differ. The checkpoints use a custom tensor type, so they only work with this project's launcher or binary and not with mainstream tools such as Ollama or LM Studio. Faster Top-5 and Top-4 modes are lossy, with KL divergence up to 0.0694 nats in the Turbo mode. The pre-built binary is for Linux x86_64. This is a solo GitHub project with no author name or institution given beyond the account name, and the speed claims should be reproduced before they are relied on.