LookAtMyQuant shrinks Inkling-2-Mini to a 5.4 GB Apple Silicon build

LookAtMyQuant shrinks Inkling-2-Mini to a 5.4 GB Apple Silicon build

LookAtMyQuant is an open-source quantization compiler released under the MIT license. It measures how much each stored piece of a model hurts the output when compressed, then picks a code for each piece so the whole model fits a size budget with the least damage. The README says that on Gemma 4 E2B its GGUF has 12x lower KL than Q4_K_M at the same file size.

The headline result is for thinkingmachines/Inkling-2-Mini, described as a 26.4B physical / 5.4B active 3x-looped MoE (128 experts, top-8). The compiler plus full-MoE proximal fine-tuning (SQYFR) compress its 18.89 GB of text weights into four Apple Silicon MLX builds: 5.38 GB (1.65 bits per weight), 8.87 GB (2.72 bpw), 11.18 GB (3.43 bpw) and 12.76 GB (3.92 bpw). The recipe combines hashed block-vector trellis codes, row trellis codes, a floor on the tail lm_head bucket, and exact native NVFP4 passthrough for the most sensitive expert buckets in the two largest builds. The vision tower stays untouched at BF16 (637 MB). Note that 5.4 refers to the smallest file's size in GB; the 5.4B in the model description is its active parameter count, and the 5.4 GB build needs 5.98 GB of active RAM when running.

Evaluation was untruncated (max_new_tokens 1,536, greedy, reasoning_effort set to none) against the same BF16 teacher, on GSM8K test[:200], all 164 executable HumanEval problems, four vision VQA tests and 16+13-language text windows. The 12.76 GB build matches 100.0% of the BF16 teacher on every benchmark (349 / 349 pooled) with 32.5% less text weight memory. The 11.18 GB and 8.87 GB builds both retain 99.2 to 99.3% on the 3-task mean (345 / 349 = 98.9% pooled). The 8.87 GB build cuts KL 2.34x on English, chat and code (0.0208 vs 0.0486) and 29.6% across all 13 languages (0.0571 vs 0.0811) compared with Unsloth's 9.14 GB UD-Q2_K_XL GGUF. The 5.38 GB build scores 88.0% on GSM8K and 70.7% on HumanEval (105/164 = 64.0% under the strict content_text reading), and the README says it also beats Unsloth's 7.76 GB UD-IQ1_S and 9.14 GB UD-Q2_K_XL under an identical 384-token harness.

Speed is reported on an Apple M5 Max, running through the project's lamq command and runtime_mlx with single-launch Metal kernels, optionally with 178.9 MB 4-bit DSpark drafters for lossless speculative decoding. The 12.8 GB build reaches 2,016 tok/s prefill (T=512), 60.5 tok/s serial and 112.7 tok/s median drafted decode (12.72 GB active RAM). The 11.2 GB build reaches 2,032 tok/s prefill, 62.4 tok/s serial and 103.7 tok/s median (11.13 GB active RAM). The 8.9 GB build decodes at 63.5 tok/s serial and 106.0 tok/s median (9.41 GB active RAM). The 5.4 GB build reaches 1,807 tok/s prefill, 75.3 tok/s serial and 124.4 tok/s median, with 142.9 tok/s on math, 126.5 on reasoning and 122.2 on code (5.98 GB active RAM).

The README also covers earlier work with the same method. LookAtMyQuant 2 takes Qwen3.5-4B to 1.23 GB (2.3 bpw). That file solves 330 of 540 exact-answer math problems, against 328 for Prism ML's Bonsai 2 Ternary 4B at 1.20 GB and 359 for the original; its KL from the original on held-out text is 0.45 nats against 0.59 for Bonsai. The recipe is post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10x, with bits allocated per row by Fisher information, followed by 4,000 steps of distillation. It costs about $34 of GPU time. The same recipe takes Qwen3.8-27B from 53.7 GB to 7.2 GB (2.1 bpw). That file solves 401 of the 540 math problems, as many as the original, where Unsloth's UD-IQ2_XXS GGUF of the same size solves 348. It also scores higher than that GGUF on HumanEval (73.8% vs 67.1%), MMLU (78.8% vs 75.0%) and GSM8K (94.5% vs 92.7%). A 1.02 GB 4-bit DSpine drafter, converted from tencent/Qwen3.8-27B-DSpine-b16, roughly doubles decode speed on math and code on Apple silicon.

For reproduction, the README lists two recent zero-byte recipe additions: Softmax-Fisher token table ranking and mid-training equal-byte row reallocation. Combined on Qwen3.5-4B at an identical 1.23 GB, they lowered WikiText-2 KL from 0.425 to 0.387 (a 9.0% drop) and 13-source KL from 0.447 to 0.378 (a 15.5% drop). It also walks through measuring, allocating and scoring a quantization plan on SmolLM2-360M (a 3.044 bits-per-weight plan reached perplexity 16.64 and KL 0.0926 against an original perplexity of 15.31), and gives Modal job commands to rebuild the Qwen3.5-4B file.

Key facts

  • LookAtMyQuant, an MIT-licensed quantization compiler, compresses thinkingmachines/Inkling-2-Mini from 18.89 GB of text weights into four Apple Silicon MLX builds of 5.38, 8.87, 11.18 and 12.76 GB.
  • The 12.76 GB build matches the BF16 teacher on all benchmarks (349 / 349 pooled) with 32.5% less text weight memory; the 5.38 GB build scores 88.0% on GSM8K and 70.7% on HumanEval.
  • The 8.87 GB build cuts KL 2.34x on English, chat and code versus Unsloth's 9.14 GB UD-Q2_K_XL GGUF (0.0208 vs 0.0486).
  • On an Apple M5 Max with 4-bit DSpark drafters, the 5.4 GB build reaches 124.4 tok/s median decode (75.3 tok/s serial) and needs 5.98 GB of active RAM.
  • The same recipe takes Qwen3.5-4B to 1.23 GB (330 of 540 math problems solved, vs 359 for the original) and Qwen3.8-27B from 53.7 GB to 7.2 GB.

Why it matters

A 26.4B-parameter MoE that starts at 18.89 GB of text weights ends up as a 5.38 GB file, while the README reports 88.0% on GSM8K and 70.7% on HumanEval for it. The larger builds report near-parity with the BF16 original: the 12.76 GB build matches it on all 349 pooled items. The README also claims the 8.87 GB build beats a similarly sized Unsloth GGUF on KL by 2.34x on English, chat and code, so the pitch is better quality at the same size, not only smaller files.

Who it affects

People running large models locally on Apple Silicon, since all four Inkling-2-Mini builds target the MLX runtime and the speeds are measured on an Apple M5 Max. It also concerns anyone working on quantization methods or comparing against Unsloth GGUFs, Q4_K_M and Prism ML's Bonsai 2 Ternary 4B.

How to use it

On Apple silicon, install the runtime with pip install git+https://github.com/ShinMegamiBoson/lookatmyquant.git, which brings in mlx, mlx-lm, numpy and huggingface-hub without the PyTorch compiler dependencies. Then run, for example, lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-5.4GB with --drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter and a prompt; a 12.8GB repo is also listed. lamq chat and a Python API (runtime_mlx.load, Runtime) are available. The code is under the MIT license. The compiler itself can be run from the repository through its measure, allocate and quantize steps, with a worked example on SmolLM2-360M.

How solid is it

The numbers are specific and the harness is described: same BF16 teacher, GSM8K test[:200], all 164 executable HumanEval problems, four VQA tests and multilingual text windows, greedy decoding. All results are the authors' own, and the visible text mentions no independent verification or replication. The 5.38 GB build's 70.7% HumanEval drops to 64.0% (105/164) under the strict reading, so the headline figure depends on how outputs are parsed. The GSM8K set is only the first 200 test items.

Risks and caveats

The 5.4 GB figure is file size, not runtime memory: the active RAM of that build is 5.98 GB. Speed figures are on an Apple M5 Max and the median drafted numbers use the 4-bit DSpark drafter; serial speed without it is 75.3 tok/s for the smallest build. The visible text does not say whether the Inkling-2-Mini builds run on non-Apple hardware or in llama.cpp. The evaluation used reasoning_effort set to none, so the results say little about thinking mode. The README itself notes that its own 13-source text KL for Qwen3.5-4B comes out at 0.44 rather than the page's 0.45 when run with the mix's own evaluation windows.