Glonce

Puro-2B recipe trains a 2B model for under $6.9K on RTX 5090s

researchThe Glonce editorsHugging Face🔥73Glonce rating73Impact65Novelty68Relevance100Surprise60Players35Credibility88
Puro-2B recipe trains a 2B model for under $6.9K on RTX 5090s

Researchers have released Puro-2B, a collection of language models pretrained from scratch on consumer-grade RTX 5090 GPUs, along with the full recipe behind it: data, code and model weights, published under an Apache 2.0 license on a Hugging Face collection. The paper frames the work as a response to how expensive pretraining has become. As baseline examples, it cites training Llama-3.2-3B at over $1.5M and reproducing SmolLM3-3B at over 700K, costs the authors say put pretraining out of reach for most academic and open-source groups even at small model scale. Puro-2B models were trained on up to 1.4 trillion tokens using FP8 precision. The collection's best model, the one actually trained rather than projected, reached a compute cost of less than $6.9K and approaches, without matching or exceeding, the performance of Qwen2.5-1.5B under the authors' own evaluation protocol. The authors attribute this efficiency to a combination of methods rather than any single trick: hardware selection, low-precision training, an optimization approach they call hyperball optimization, curriculum model averaging, and the data recipe itself. From results across the whole Puro-2B collection, the authors fit what they call a Puro Cost Scaling Law, relating training cost to average model performance. Extrapolating from that law, they estimate that about $4.4K, under $5,090, would be enough to reach the performance of Qwen2-1.5B, a separate, older baseline model from the one the actually trained model was compared against. The paper also reports a second result beyond the recipe: a case study on how pretraining data curricula shape downstream performance after post-training, a study the authors say is only possible because they have access to the full pretraining pipeline rather than just released model weights.

Key facts

  • Puro-2B models were pretrained from scratch on up to 1.4 trillion tokens using FP8 precision on consumer RTX 5090 GPUs.
  • The best model actually trained cost under $6.9K in compute and approaches, but does not match or exceed, Qwen2.5-1.5B performance under the authors' own evaluation protocol.
  • A fitted Puro Cost Scaling Law extrapolates that about $4.4K, under $5,090, would be enough to reach Qwen2-1.5B performance, a separate baseline from the one the trained model was measured against.
  • The paper cites baseline pretraining costs of over $1.5M for Llama-3.2-3B and over 700K for reproducing SmolLM3-3B as context for how expensive pretraining normally is.
  • The full recipe, data, code and model weights are released under Apache 2.0 on a Hugging Face collection, enabling a separate case study on how data curricula affect downstream performance after post-training.

Why it matters

Pretraining a language model from scratch has been treated as something only well-funded labs can afford, with the paper's own cited baselines running into the hundreds of thousands or millions of dollars even for small, few-billion-parameter models. Puro-2B is pitched as a full, reproducible counterexample: not just a released model, but the complete recipe, on hardware anyone can buy. Cutting the demonstrated cost of a competitive small model down to under $6.9K, with a scaling law suggesting under $5,090 is achievable, reframes pretraining as something a well-resourced individual, lab, or open-source project could realistically attempt rather than something reserved for large compute budgets.

Who it affects

The paper frames its audience explicitly: academic researchers and open-source contributors who have been priced out of pretraining experiments and have had to rely on models and recipes released by others. Anyone doing research on pretraining dynamics, data curricula, or optimization at small scale stands to benefit most, since the release includes not just weights but the data and code needed to rerun or modify the process, which the authors note is what let them run their own downstream case study.

How to use it

The full training recipe, including data, code, and model weights for the Puro-2B collection, is released under the Apache 2.0 license on a Hugging Face collection. That license permits reuse, modification, and commercial use with attribution. Because the release includes the pipeline itself and not only trained weights, it is set up to be rerun or adapted on similar consumer GPU hardware rather than simply downloaded and deployed as a finished chat model.

How solid is it

The source text is the paper's own abstract, and it is explicit about the distinction that matters most here: the $6.9K figure is the cost of a model the authors actually trained and evaluated against Qwen2.5-1.5B under their own protocol, while the $4.4K and under $5,090 figures come from extrapolating a fitted cost scaling law toward a different baseline, Qwen2-1.5B, rather than from an actual training run. The abstract does not include a benchmark table or specific accuracy numbers, does not state how many RTX 5090 GPUs were used, and does not give a training wall clock time, so the performance claims rest on the authors' own evaluation protocol as described rather than on externally reproduced figures.

Risks and caveats

The two headline cost figures answer different questions and should not be collapsed into one: under $6.9K is an actual training cost measured against Qwen2.5-1.5B, while under $5,090 is a scaling law's projection toward the separate, older Qwen2-1.5B baseline. The abstract also does not name the paper's authors or institutions, and it does not specify GPU count or training time, so independent verification of the exact setup is not possible from the source text alone. The baseline comparison costs (over $1.5M for Llama-3.2-3B and over 700K for SmolLM3-3B) come from the authors' own framing rather than from an independent audit of those other projects' budgets.