Glonce

BitNet-quantized Mamba model runs on a 1975 MOS 6502 chip

researchThe Glonce editorsHacker News🔥50Glonce rating50Impact40Novelty65Relevance88Surprise72Players18Credibility82

The author wanted to find the strongest language model that could run on a BBC Micro from the 1980s, using their father's original machine. The BBC Micro is built around the MOS 6502, an 8-bit microprocessor released in 1975 that also powered the Apple II. The constraint was severe: model weights and inference code together had to fit in 25KB of user-space memory, out of the machine's 32KB of total RAM. The final split was 9KB of inference code and 13KB of model weights. The 6502 works only with 8-bit integers and has no multiply instruction at all.

The inference code was written in C and compiled to 6502 machine code with the CC65 toolchain. The resulting binary was loaded onto the real BBC Micro through PlayUEF and a homemade 3.5mm-to-tape cable, which makes the machine behave as though it is reading a tape drive while a laptop plays the program out through its headphone jack. Correctness was checked with the sim65 emulator, comparing the C inference binary against a reference Python implementation of the model, and the full inference pipeline was tested in the jsbeeb emulator before ever touching the physical hardware. Readers can run the demo themselves: a link boots a BBC Micro emulator in the browser, loads the tape image straight from GitHub, and auto-types the commands needed to run the model, no local emulator install required, though generation takes a few minutes.

The model is autoregressive, producing one token at a time the way larger language models do. Its vocabulary is deliberately tiny, just the 26 letters of the alphabet plus a space character, because a larger word- or subword-level vocabulary would consume too much of the available parameter budget on its own. An embedding layer maps each token into a hidden dimension of 56.

Weight compression comes from BitNet, a quantization method built for fast CPU inference. BitNet constrains every weight to the ternary set {-1, 0, 1}, which turns a matrix multiplication into a sequence of additions and subtractions instead of multiplications. Each BitNet parameter then needs only about 1.58 bits of storage (log2 of 3), against 8 bits for int8 or 32 bits for float32. On the 6502 this matters directly: a standard 8x8 multiply-and-accumulate costs 150 clock cycles, while a ternary accumulate costs only 30. Four ternary parameters are packed into each byte; a 5-per-byte packing would be more space-efficient but was rejected because unpacking it needs a repeated floor-divide-by-3 operation that the 6502 cannot do natively, while 4-per-byte unpacking is just a right-shift. At 4 parameters per byte, the 13KB weight budget holds about 52,000 BitNet parameters. During training, weights are kept in full float32 precision, quantized to ternary on the forward pass, and given a straight-through gradient estimator so full-precision gradients still flow on the backward pass. The one exception is the final output (LM head) layer, kept at int4 rather than ternary, because spreading probability cleanly across the vocabulary needs more resolution; every other weight matrix in the model is ternary.

Choosing the model architecture meant ruling out two standard options. Attention, as used in GPT-3-style transformers, was rejected because its KV cache grows with every token generated, and that growth would eat into the already scarce 32KB of RAM. A GRU-based recurrent model was also rejected: every training run diverged. The instability traces to the spectral radius of the forward weight matrices, which in the BitNet regime typically stays far above 1; getting it down to roughly 1 would require about 98% of the weights to be zero. The only workaround found was storing the GRU's primary weight matrix at a higher quantization such as int4, but since that is the largest matrix in the model, doing so would sharply shrink the achievable model size. Mamba, a state-space model, avoids the problem by construction: its per-channel decay value is computed at inference time and constrained to the range [0, 128)/128, so it can never exceed 1 and the runaway-eigenvalue failure mode cannot occur. That made Mamba the chosen architecture.

Activations are stored in 8 bit, and each accumulator term is capped at 128 in magnitude, so up to 256 terms can be summed into 16 bit without overflow before being clipped back down for the next layer. A naive clip straight to 8 bit throws away most of the dynamic range: accumulating 64 values drawn roughly uniformly from -128 to 127 produces a standard deviation of about 591, meaning 83% of values would saturate at the -127 or 128 clip bounds. Instead the model uses a learned right-shift scaling parameter before clipping, and that scale is allowed to vary during the first half of training before being frozen for the second half.

Sampling posed its own problem: greedy sampling is a simple max over logits, but top-k softmax sampling needs an exponential function the 6502 cannot compute natively. The workaround is a precomputed lookup table of exponential values keyed to a chosen temperature, combined with the usual softmax stability trick of subtracting the maximum logit before the lookup; the final token is then chosen with a pseudo-random 16-bit integer taken modulo the sum of the table's values. Because the 6502 offers no way to seed that randomness from outside without user input, the seed is fixed, so the model produces the same generated output on every run.

The end result is a Mamba-based model that genuinely runs inference on the physical BBC Micro. The author is candid that it 'isn't particularly intelligent,' but treats the working demonstration itself, a model architecturally similar to contemporary language models running on 1980s hardware, as the point. The write-up frames the project as an exercise in mechanical sympathy, designing modeling choices around the target hardware, and ties it to Sarah Hooker's hardware lottery thesis: that state-of-the-art architectures and algorithms have evolved alongside the hardware available to run them, implying a different set of hardware constraints could have favored a different architecture or optimizer entirely.

Key facts

  • The author fit a BitNet-quantized, autoregressive Mamba language model into 25KB of user-space memory (9KB inference code, 13KB weights) and ran it on a real BBC Micro built around the 1975 MOS 6502 chip, which has 32KB of RAM and no multiply instruction.
  • BitNet quantizes weights to the ternary set {-1, 0, 1}, needing about 1.58 bits per parameter versus 8 bits for int8 or 32 for float32; on the 6502 a ternary accumulate takes 30 clock cycles versus 150 for a standard 8x8 multiply-accumulate, and 4 parameters are packed per byte, fitting about 52,000 parameters into 13KB.
  • Mamba was chosen over attention, whose growing KV cache would exhaust the 32KB RAM budget, and over GRUs, which diverged in every training run because their forward matrices need about 98% zero weights to keep the spectral radius near 1; Mamba's per-channel decay value is bounded to [0, 128)/128 by construction, ruling out that instability.
  • Because the 6502 has no way to inject a random seed without user input, the model's token sampling is deterministic and produces the same generated text on every run; softmax itself is approximated with a precomputed exponential lookup table rather than computed directly.
  • The write-up frames the project as an example of Sarah Hooker's 'hardware lottery' thesis, that state-of-the-art architectures have evolved alongside the hardware available to run them, and offers a browser demo that boots a BBC Micro emulator and auto-runs the model with no local install.

Why it matters

The project is a working demonstration of how far weight compression can go: a BitNet-quantized model with roughly 52,000 parameters, running inference with real add/subtract arithmetic on a chip with no multiply instruction and 32KB of RAM. It also makes a concrete case for the 'hardware lottery' idea the author cites: the choice between attention, GRUs and Mamba here was decided almost entirely by memory and instability constraints specific to the 6502, not by which architecture is generally considered strongest.

Who it affects

This is a hobbyist and retrocomputing project rather than a product aimed at any user base. It is most relevant to people interested in extreme quantization, edge and embedded machine learning, or the history of the hardware that shaped early machine learning research, plus anyone curious about running modern model architectures on 1970s and 1980s silicon.

How to use it

There is nothing to buy or install to try it. A link in the post boots a BBC Micro emulator directly in the browser, automatically loads the tape image from GitHub, and auto-types the commands needed to run the model; generation takes a few minutes and no local emulator setup is required. Readers wanting to inspect or reuse the approach can also look at the inference building blocks referenced in the post.

How solid is it

The claims are backed by a documented, checkable pipeline rather than a bare assertion: the C inference code is validated for parity against a reference Python implementation using the sim65 emulator, and the full inference engine is tested in the jsbeeb emulator before being run on the physical BBC Micro. The post also works through the underlying math (bit budgets, cycle counts, spectral radius, accumulator overflow) in enough detail to check the reasoning, though it remains a single-author write-up rather than a peer-reviewed result.

Risks and caveats

The model operates at toy scale: a 26-letter-plus-space vocabulary, a 56-dimension embedding, and around 52,000 parameters, and the author states directly that it 'isn't particularly intelligent.' Because the 6502 offers no way to seed randomness without user input, the sampling is deterministic and the model generates identical output on every run, which is a hardware limitation rather than a property of the modeling approach. No benchmark comparison against any other language model is given, and the article does not name the training dataset or its size.

“While it isn't particularly intelligent, it does demonstrate running a model similar to contemporary models on hardware from the 1980s. Pretty awesome!”

— the author, in the write-up on mattbeton.com