DeepSeek-V3 architecture rebuilt in 190 lines of pure JAX

DeepSeek-V3 architecture rebuilt in 190 lines of pure JAX

A GitHub repository called deepseek-v3-jax, posted to Show HN, implements the architecture from the DeepSeek-V3 technical report in 190 lines of pure JAX. The Show HN title rounds this to about 200 lines. There is no Flax, no Equinox and no PyTorch: parameters are plain dicts of arrays, and every part of the model is a function that can be read top to bottom in deepseek_v3.py. The 190 figure counts the non-blank, non-comment lines of that file after its docstring, and the README gives the shell one-liner that reproduces it.

The author pretrained a small model from scratch on a MacBook Air M5 with 24 GB of memory. The training text is public-domain detective fiction: Dashiell Hammett's Red Harvest, The Dain Curse, The Maltese Falcon and the Continental Op stories (CC0 editions from Standard Ebooks), plus about 200 detective and crime novels from Project Gutenberg. In all, prepare_data.py downloads 207 books listed in sources.txt (about 120 MB, roughly 7 minutes), cleans them and trains an 8,192-token BPE tokenizer. It should print a training split of 211 documents, 83.4M characters and 21.69M tokens, and a validation split of 4 documents, 1.1M characters and 0.30M tokens.

The model is tiny next to the real thing: 36.2M parameters versus 671B, 6 layers versus 61, and a 256-token context versus 128K. The main run is 12k steps and takes about 2.6 hours. The README shows one sample, generated from the prompt "The fog came in off the": "The fog came in off the bay, and the great trees were spreading and the stars were shining." Its own verdict is plain: this is an architecture demo, not a strong language model. A separate write-up, with diagrams and the code explained piece by piece, covers what worked, what did not and what is missing.

One experiment is reported in the README. Two 2000-step runs, one with the load-balancing bias on and one with it off, ended with the same validation loss at step 2000: 3.921 with balancing, 3.923 without. The README reads this as the bias evening out the experts without costing anything, which it describes as the paper's claim.

The README lists what the code leaves out. MoE compute is dense: every expert runs on every token and the unpicked ones are zeroed. The routing math is exact, but there is no sparse dispatch or expert parallelism. There is no weight absorption in multi-head latent attention (MLA), so the latent is decompressed explicitly, which is easier to read and slower at inference. FP8 training, DualPipe, node-limited routing, YaRN context extension and any post-training are not included.

For reproduction, you need an Apple Silicon Mac, Python 3.12 and uv. The v1.0 release carries the exact checkpoint (138 MB), the tokenizer and the data from the published run. Without jax-metal everything still runs, only much slower, and a smaller config (4 layers, 8 experts, top-2, 128-token context) is offered for 8 to 16 GB Macs. The code is MIT licensed.

Key facts

  • The DeepSeek-V3 architecture is implemented in 190 lines of pure JAX (the Show HN title says ~200), with no Flax, Equinox or PyTorch.
  • A 36.2M-parameter model (6 layers, 256-token context) was pretrained from scratch on a MacBook Air M5 with 24 GB, on public-domain detective fiction; the main run is 12k steps, about 2.6 hours.
  • In 2000-step ablations, validation loss was 3.921 with the balancing bias and 3.923 without it.
  • Simplifications include dense MoE compute, no MLA weight absorption, and no FP8, DualPipe, node-limited routing, YaRN or post-training.
  • The README calls the project an architecture demo, not a strong language model; code is MIT, training text is public domain in the US.

Why it matters

The DeepSeek-V3 architecture is usually met as a technical report and a 671B-parameter model. This repository squeezes it into a file short enough to read in one sitting, with parameters as plain dicts and each model part as a function. It is a learning aid rather than a new model: the real model has 61 layers and a 128K context, while this one has 6 layers and a 256-token context. The ablation also gives a small, concrete check of one paper claim, that the balancing bias evens out the experts without hurting validation loss.

Who it affects

Mainly people who want to understand how DeepSeek-V3 is built by reading and running code: students, and engineers curious about mixture-of-experts and latent attention. It also suits anyone with an Apple Silicon Mac who wants to train something end to end on a laptop. It is not aimed at people looking for a usable language model.

How to use it

Clone the repository, create a Python 3.12 environment with uv and install the pinned requirements. Run the tests on CPU (the README expects 39 passed), then run prepare_data.py to build the dataset, or download the published data from the v1.0 release to train on the exact same bytes. Run train.py for the main run (12k steps, about 2.6 hours), or train.py with --steps 2000 and with or without --no-bias for the ablation. Generate text with sample.py (defaults: temperature 0.8, top-k 50, 300 new tokens per sample). The code is MIT licensed. The training text is public domain in the United States, and the README notes that status outside the US may differ.

How solid is it

The repository documents its setup in detail: pinned versions, expected dataset sizes, hashes for the published data and 39 tests. The README says its own results should land close to, not exactly match, the published numbers, because GPU math on Metal is not bit-for-bit deterministic. The bias ablation is a single pair of 2000-step runs; the source does not say it was repeated across seeds. The README's results block for the main 12k-step run is empty in the text provided, so no final loss or throughput figure is available here. The only output quality evidence is a sample sentence, and the README itself says this is not a strong language model.

Risks and caveats

The checkpoint is a Python pickle, which can run code when loaded, so load checkpoints only from sources you trust. jax-metal 0.1.1 only works with jax==0.4.34, so JAX should not be upgraded. One check in smoke_metal.py, grad through a 2-axis contraction, fails on jax-metal 0.1.1; the README says that is expected and the model is written to avoid it. Regenerated data will most likely give different hashes: Project Gutenberg has since re-issued 28 of the 203 books with small edits, though token counts come out the same. The smaller config has only been checked for a couple of steps, not trained to the end. Dense MoE compute and explicit latent decompression make the code easier to read but are not how the full model would be run efficiently.

“This is an architecture demo, not a strong language model.”

— the project README