Weight-space programming: LLMs writing weights by hand pick tables, not Transformers

A paper on the Hugging Face papers page looks at what it calls weight-space programming: a large language model writes a network's target weights by hand, without gradient-based training. The authors start from a simple observation. Standard architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs. If no gradient training happens, those reasons no longer apply, so they ask: should we keep the Transformer?
To test it, the authors task three frontier LLMs, mainly on multiple-choice questions. Each model gets minimal guidance, ten labeled examples and a sandbox. In one setup the LLM generates the weights of a fixed architecture. In the other it designs the architecture itself.
When free to design, all three LLMs make the same choice. They compile plain-text facts into a sparse table that is queried by a fixed procedure, reaching up to 54.4% test accuracy. One LLM-designed model achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote, while using about 1,800x fewer parameters.
The authors explain the choice by interpretability and repair. In a table, the LLM can state what each weight means and can find and rewrite a wrong one. Each wrong answer points to one row, and removing a row changes about one other answer. Without gradients, the models settle on a structure they can read and edit.
From this, the authors propose an associative-memory-based model that is up to 17x faster on GPUs than a same-sized Transformer. They also test pretrained weights, other benchmarks, and tasks where models generate answers.
The authors limit their claim. Pretrained LLMs remain more accurate; the claim concerns which structure to build when writing weights, not which model is better overall.
Key facts
- In weight-space programming an LLM writes a model's weights by hand with no gradient training, so the usual reasons for choosing a Transformer (trains well with gradient descent, fast on GPUs) no longer apply.
- Three frontier LLMs, given minimal guidance, ten labeled examples and a sandbox, mainly on multiple-choice questions, all chose a sparse table queried by a fixed procedure when free to design, reaching up to 54.4% test accuracy.
- An LLM-designed model beat the most accurate released Transformer written by the same LLM on test accuracy, with about 1,800x fewer parameters.
- The authors propose an associative-memory-based model that is up to 17x faster on GPUs than a same-sized Transformer.
- Pretrained LLMs remain more accurate; the claim is about which structure to build when writing weights.
Why it matters
The Transformer is the default architecture largely because it suits gradient training and GPUs. This paper asks what happens when weights are written directly by an LLM instead. Its answer is that the models drop the Transformer on their own and build a readable lookup table, which suggests that the best architecture may depend on how the weights are produced. The authors tie the choice to editability: each wrong answer points to one row, and the LLM can rewrite it.
Who it affects
Mainly researchers working on model architectures, interpretability and model editing, and anyone studying whether LLMs can produce working models without training. The paper is a preprint with no direct product impact described.
How to use it
There is nothing to install or run in the abstract. The practical idea is a design choice: when weights are written by hand or by an LLM, a sparse table queried by a fixed procedure, or the associative-memory-based model the authors propose, is the structure they point to instead of a Transformer.
How solid is it
The results are the authors' own experiments and the source is an abstract. The 54.4% figure is a maximum ('up to'), not an average across LLMs or tasks. The 1,800x parameter reduction is measured against the most accurate released Transformer written by the same LLM, not against pretrained production LLMs. The 17x speedup is also a maximum, for the proposed associative-memory-based model against a same-sized Transformer. The abstract does not name the three frontier LLMs. The abstract does not name the benchmark or datasets, nor the baseline test accuracy of the Transformer it was compared against.
Risks and caveats
The authors say plainly that pretrained LLMs remain more accurate, so this is not a claim that tables beat Transformers in general. The tests rest mainly on multiple-choice questions with ten labeled examples. The authors say they also test pretrained weights, other benchmarks and tasks where models generate answers, but the abstract does not give those results. The abstract does not say whether code or weights are released.
“Pretrained LLMs remain more accurate; the claim concerns which structure to build when writing weights.”
— Paper abstract, Hugging Face papers