WASTE engine streams full 2.78T-parameter Kimi K3 from disk

WASTE is an embeddable inference engine written in C with no third party runtime dependencies. Its proof point is Kimi K3, an open weights mixture of experts model with 2.78 trillion parameters. Kimi K3 is 1.42 TB as published; WASTE converts it into a 982 GiB container and runs the complete, undistilled model on a 64 GB MacBook Pro at 0.45 to 0.62 tokens per second. In the project's own demo, a 16 token answer to "What is the capital of Italy?" took 25.78 seconds.

The trick is architectural rather than a smaller model. A mixture of experts model activates only about 4% of its weights per token, so WASTE keeps the model's resident trunk, 27.28 GB, in RAM and streams the selected experts straight from disk for each token, bypassing the OS page cache (F_NOCACHE on macOS, O_DIRECT on Linux, FILE_FLAG_NO_BUFFERING on Windows) so the measured hit rates reflect real disk behavior rather than an artifact of a container smaller than RAM getting cached. On the default 64 GB machine, the engine budgets 46 GB, of which 17.56 GB is expert cache; a single token's forward pass reads 17 GB of experts, so storage speed matters directly: 12.78 GB/s on an internal SSD keeps the model streaming, while a USB enclosure at 0.94 GB/s stretches the same token to about 13 seconds.

Correctness is validated layer by layer against a PyTorch reference, with final output logits agreeing to 3.6e-06 and the vision tower matching its own oracle to 2.3e-06. Two measured levers account for the practical speed: overlapping expert disk reads with the arithmetic is worth about 1.6x, and a router-lookahead trick, running the next layer's router one residual early on the current layer's hidden state and prefetching its top experts, raises the expert-cache demand hit rate from 14% to 38% while reading the same total bytes; the speculative guess is right 92% of the time at rank one and 81% over the first six. The authors say the technique is exact by construction: the real router still makes the final call, the speculative fetch only changes when bytes move.

Not every idea survived. A 3-bit trunk quantization was built and measured: the cache prediction still worked, but throughput did not improve and output quality collapsed, so the trunk stays at 4 and 8 bits, with experts stored via residual vector quantization at 3.00 bits per weight. A speculative-fetch variant tried in the prefill path was also removed after measurement showed it cost 6.9% more bytes for no time saved. The project's own benchmark table shows how fragile these numbers get near the memory boundary: runs at 32 GB and 46 GB reproduce within a few percent, but five runs at a 52 GB budget spread from 0.03 to 0.46 tokens per second, a fifteen-fold range, while reporting identical cache hit and miss counts every time, because that budget sits exactly where the engine's footprint does or does not fit beside whatever else is running. Measuring right after a 52 GB or 58 GB run leaves the machine paging and can make even a healthy 46 GB budget read as low as 0.02 tokens per second.

The authors say they are not aware of another published demonstration of a model this size streaming from disk on a consumer machine, and that the best documented recipes for 671B-class models they know of assume a server with a terabyte of DDR5, but they are explicit that this reflects what their own search turned up rather than a systematic survey, since the repository carries no bibliography or comparison table. The format and engine are not specific to K3: the same code also runs the far smaller Kimi-Linear-48B-A3B-Instruct model from a 19 GB container with a 1.87 GB RAM floor at 10.7 tokens per second, offered as an easier way to try WASTE before committing a disk to the full K3 model.

Key facts

WASTE, an embeddable C inference engine, runs the full 2.78 trillion parameter open weights Kimi K3 model on a 64 GB MacBook Pro at 0.45 to 0.62 tokens per second by keeping the resident trunk in RAM and streaming mixture of experts weights from disk.
Kimi K3 is 1.42 TB as published and converts into a 982 GiB container; the resident trunk alone needs 27.28 GB, and the default 64 GB machine budget of 46 GB includes 17.56 GB of expert cache, with a single token reading 17 GB of experts from disk.
Correctness is validated against a PyTorch reference: final logits agree to 3.6e-06 and vision-tower output agrees to 2.3e-06.
Two engineering levers drove the practical speed: overlapping expert disk reads with arithmetic is worth about 1.6x, and a router-lookahead prefetch, guessing the next layer's experts one residual early, raises the expert-cache hit rate from 14% to 38% while reading the same total bytes.
The same engine and container format also run the smaller Kimi-Linear-48B-A3B-Instruct model from a 19 GB container at 10.7 tokens per second, offered as an easier way to try WASTE before committing a disk to the full K3 model.

Why it matters

Kimi K3 is a 2.78 trillion parameter model that the authors say the best documented 671B-class recipes they know of would put on a server with a terabyte of DDR5; WASTE demonstrates the full, undistilled model answering prompts on a 64 GB consumer MacBook Pro by treating idle expert weights as data to fetch rather than data to hold in memory. The authors frame this as a shift from feasibility to engineering: once a frontier-scale model is reachable at all on ordinary hardware, the remaining question is how fast, not whether. They say they know of no other published demonstration of trillion-scale streaming from disk on consumer hardware, though they explicitly flag that claim as a report of their own search rather than a systematic survey, since the repository carries no bibliography or comparison table.

Who it affects

Developers who want to run large open-weights models locally without a cloud API bill, without network access, or without sending data off the machine; the project's own framing is the difference between "you may not send that data to an API" and "run it here." It also matters to anyone building or evaluating mixture-of-experts inference systems, since the levers involved, expert caching, read-ahead prefetch, and a disk layout where each expert costs exactly one read, are general techniques rather than Kimi-K3-specific ones; the same engine and container format already run the smaller Kimi-Linear-48B-A3B-Instruct model.

How to use it

WASTE ships as a CLI (waste run "") and as an embeddable C API with 26 public functions covering opening a model under a RAM ceiling, generating, saving a session, and closing; the CLI is described as just a client of that same API, so an embedding host can do anything the CLI can. Running the full Kimi K3 requires converting the 1.42 TB published weights into a 982 GiB container and having at least 64 GB of RAM, since a 32 GB machine can technically open the model but pages badly. The container should sit on internal NVMe storage rather than a USB drive, since read speed drops from 12.78 GB/s to 0.94 GB/s and a single token's read time grows from about one second to roughly 13. Readers who do not want to commit close to a terabyte of disk to K3 can instead try the same engine on Kimi-Linear-48B-A3B-Instruct, a 19 GB container with a 1.87 GB RAM floor running at 10.7 tokens per second. No pricing or licensing terms for WASTE are stated in the source.

How solid is it

The engine's correctness is checked against a PyTorch reference layer by layer, with final logits agreeing to 3.6e-06 and the vision tower matching its own oracle to 2.3e-06, and the authors say the router-lookahead prefetch is exact by construction because the real router still makes the final routing decision. The project documents its own measurement noise rather than hiding it: runs at a 52 GB budget produced throughput between 0.03 and 0.46 tokens per second, a fifteen-fold spread, while reporting identical cache hit and miss counts each time, because that budget sits exactly at the edge of whether the engine's footprint fits in the machine. The authors note that measuring one budget right after a heavily paging run can make an otherwise healthy configuration read as low as 0.02 tokens per second, and instruct readers to sweep budgets upward on a quiet machine and discard anything measured after a paging row. They also say every number in the writeup was measured on the published commit, with incorrect earlier numbers recorded as wrong rather than quietly corrected.

Risks and caveats

At 0.45 to 0.62 tokens per second the system is slow by any interactive standard; the project's own demo shows a 16-token answer taking 25.78 seconds, and the authors call this a genuine limitation rather than a disclaimer to skip past. The claim of being the first to stream a trillion-scale model from disk on consumer hardware is explicitly self-reported: the repository carries no bibliography or comparison table, and the authors describe the statement as an invitation for a counter-example rather than a settled result. Not every optimization worked: shrinking the resident trunk to 3-bit quantization was tried and the output quality collapsed, and a speculative-fetch variant in the prefill path was removed after it cost 6.9% more bytes for no speed gain. The benchmark table itself warns that results near the memory boundary, around 52 to 58 GB budgets on a 64 GB machine, are order-dependent and can be corrupted by machine state left over from a prior run, so anyone reproducing the numbers needs to control for that.

“We are not aware of another published demonstration of a model this size streaming from disk on a consumer machine.”

— WASTE project (GitHub README)