Glonce

Paper swaps a small Gemma-shaped model's embedding table for a frozen 16 MB one

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact40Novelty60Relevance95Surprise35Players5Credibility75
Paper swaps a small Gemma-shaped model's embedding table for a frozen 16 MB one

The paper starts from a size problem. In a small language model the vocabulary is the single largest part: in Gemma 3 270M the embedding table holds 168M of the 268M parameters. Quantisation methods leave the embedding in 16-bit, so once the rest of the model is compressed, the table is also most of the file.

The authors ask what that table actually buys. They replace it with one frozen table of 512-bit codes, 16 MB in size, with no trainable parameters. The table is derived once from a pretrained embedding by principal-component projection and sign binarisation. They then train the same Gemma-shaped body from scratch on the same 2.05B tokens, once with the frozen table and once with a learned one.

The learned table buys 0.20 nats of validation loss, at the price of 2.6 times the trainable parameters. On downstream benchmarks the picture changes: on LAMBADA, HellaSwag and PIQA the frozen-table model is better or tied, and only ARC-Easy favours the learned table.

The second result concerns size and speed. With a ternary BitNet b1.58 body, the whole model is a 46 MB file. It still ties the float model on LAMBADA and PIQA. On a laptop CPU it generates 615 to 740 tokens per second in 62 MB of memory, which is five to six times the speed of the float model's 8-bit GGUF in llama.cpp, at a seventh of the memory.

The frozen table also does better on two vocabulary-specific tests. It wins by 1.7 to 2.3 nats on rare words. And it accepts 10,000 new words after training at a 3% cost, where the learned table collapses by 43%. The authors say all of it is released, with the weights, the data and every evaluation answer.

Key facts

  • In Gemma 3 270M the embedding table holds 168M of 268M parameters; the paper replaces it in a Gemma-shaped body with a frozen 16 MB table of 512-bit codes and no trainable parameters.
  • Both variants were trained from scratch on the same 2.05B tokens. The learned table gains 0.20 nats of validation loss for 2.6 times the trainable parameters, yet the frozen-table model is better or tied on LAMBADA, HellaSwag and PIQA; only ARC-Easy favours the learned table.
  • With a ternary BitNet b1.58 body the whole model is a 46 MB file that still ties the float model on LAMBADA and PIQA, and runs at 615 to 740 tokens per second in 62 MB of memory on a laptop CPU.
  • The frozen table wins by 1.7 to 2.3 nats on rare words and takes 10,000 new words after training at a 3% cost, where the learned table collapses by 43%.
  • Weights, data and every evaluation answer are released.

Why it matters

In small models the vocabulary table dominates the parameter count, and in a compressed model it dominates the file as well, because quantisation leaves it in 16-bit. The paper tests whether that table needs to be learned at all. Its answer, for one Gemma-shaped body trained on 2.05B tokens, is that a frozen 16 MB table costs 0.20 nats of validation loss against the learned one but is better or tied on three of the four benchmarks reported. If that holds more widely, the largest block of a tiny model could be a fixed lookup rather than something trained.

Who it affects

People building or deploying very small language models, especially where file size and memory are the limit. The ternary result is the clearest case: a 46 MB model generating 615 to 740 tokens per second in 62 MB of memory on a laptop CPU. Anyone who needs to add vocabulary after training is also in scope, since the frozen table absorbed 10,000 new words for a 3% cost.

How to use it

The authors state that the weights, the data and every evaluation answer are released, so the results can be checked and the models run. The abstract gives no link or repository location for the release. The speed comparison is against the float model's 8-bit GGUF in llama.cpp, so that is the baseline to reproduce against.

How solid is it

This is a preprint abstract on Hugging Face Papers, and the claims come from the authors. In its favour, the comparison is controlled: the same Gemma-shaped body, the same 2.05B training tokens, trained from scratch with and without the frozen table. The authors also say every evaluation answer is released, which lets others audit the benchmark claims. Only the direction of the benchmark comparisons is given, not the scores.

Risks and caveats

The results cover one Gemma-shaped body of roughly 270M parameters; no other model sizes are reported. The learned table does win on validation loss (0.20 nats) and on ARC-Easy, so the frozen table is not better across the board. The ternary model ties the float model only on LAMBADA and PIQA. The laptop CPU model is not specified, and the abstract does not say which pretrained embedding the frozen table was derived from.

“In a small language model the vocabulary is the single largest part.”

— Opening line of the paper's abstract