Glonce

little-lm 3.8B beats Karpathy's nanochat on CORE for $998

researchThe Glonce editorsHacker News🔥70Glonce rating70Impact65Novelty55Relevance100Surprise68Players25Credibility78

Hugo Vergnes, an independent developer, trained a 3.848-billion-parameter, Llama-style language model called little-lm from scratch, using a self-built, YAML-configured training framework of the same name. The project was written in the evenings, debugged on a single RTX 5090, and finished on rented B200 GPUs: the final run processed 65.3 billion tokens in 43 hours for $998 and reached a CORE score of 0.384 (precisely 0.3840). For comparison, OpenAI's 2019 GPT-2 (1.5B parameters) scores 0.2565 on the same benchmark, and Andrej Karpathy's nanochat d32 (about 1B parameters, 8 H100 GPUs, roughly 33 hours) scores 0.310 for about $1,000, so at close to the same cost, little-lm's result lands clearly ahead of it. A cheaper variant of the same run, using a 1024-token context instead of 2048, reached 0.338 for $820. The project was directly inspired by nanochat, and several of its design choices, including the ClimbMix training data, BOS-aligned document packing, and the AdamW/Muon learning-rate scaling rule, were adopted from Karpathy's results rather than independently tested.

Architecturally the model follows Llama-style conventions: RMSNorm, RoPE, grouped-query attention with 24 query heads and 8 key/value heads, non-gated relu-squared MLPs, QK-norm, a logit softcap, per-layer learnable residual scalars, and ResFormer-style value embeddings. Of the 3.848B parameters, the 28 decoder layers account for 2,818.7M, the token embeddings and the untied LM head 154.5M each, and 14 tables of value embeddings 721.2M, or 19% of the total.

An earlier attempt at an 858M-parameter model, trained on FineWeb-Edu for 16.4B tokens over 5.8 days on a single A100 with AdamW at a peak learning rate of 2.5e-4 and cosine decay, scored only 60.45% on PIQA, worse than GPT-2 124M's roughly 63% despite being seven times larger and six days of compute; its loss curve had gone flat after about 70% of training. A post-mortem produced five changes for the successful run: a trapezoidal learning-rate schedule (5% warmup, a flat hold, then linear cooldown over the final 50% of steps down to 5% of peak) that kept the loss descending to the very end; the Muon optimizer for matrix parameters, whose Newton-Schulz orthogonalization step is about 25% slower per optimizer step than AdamW alone but dilutes to roughly 4% once spread across seven gradient-accumulation steps; the ClimbMix dataset in place of FineWeb-Edu, which produced a large jump in convergence speed; FP8 training on all three GEMMs plus padding the vocabulary from 50,257 to 50,304 tokens for tensor-core alignment, together worth about 33% more throughput; and, for this run, cutting context from 2048 to 1024 tokens to roughly double the batch size at fixed memory.

Separately, on the RTX 5090, the developer pushed the 858M model's throughput from a baseline of 26,144 tokens per second up to 37,621: FP8 alone added 25%, vocabulary padding took the cumulative gain to 33%, and fusing the LM head's matmul into the cross-entropy loss (Liger's FusedLinearCrossEntropyLoss) took it to 44% cumulative despite being 6% slower per step in isolation, because it freed 8GB of VRAM that let the batch size grow further; Claude initially rejected the change for measuring 6% slower, before it turned out to be a good way to claw back throughput anyway. Dropping the MLP's gate projection (SwiGLU to relu-squared) raised the small model's throughput from 183,035 to 214,173 tokens per second and saved 6GB of VRAM, though the standard SwiGLU intermediate ratio of 2.75 did not transfer to the ungated version and had to be replaced with 4x. On a separate 1.5B-parameter configuration, keeping optimizer master weights in bf16 instead of fp32 cut VRAM by 27% and raised throughput from 640K to 1.4M tokens per second, a 2.2x speed-up, at a small quality cost (CORE 0.22 versus 0.23 at 4,000 steps). Moving the same code from the RTX 5090 to a B200, on a separate 150M-parameter model in FP8, took throughput from 184,662 to 477,440 tokens per second, a 2.59x gain from hardware alone.

Several other ideas were tried and abandoned. Document-boundary attention masking, built with flex attention so tokens could not attend across packed document boundaries, was replaced with simpler best-fit, BOS-aligned packing in about 10 lines, following Karpathy's own finding that cross-document leakage does not hurt much under that packing scheme. Liger's RoPE kernel ran 2.2x faster in isolation but made no measurable difference end to end, and Liger's RMSNorm was actually slower than PyTorch 2.9's built-in implementation (0.41ms versus 0.25ms), so both were reverted. A nanochat-style initialization scheme, with embeddings at N(0, 0.8) and an LM head at N(0, 0.001), produced no measurable quality difference against GPT-2's simpler N(0, 0.02) everywhere and was kept for aesthetics only. Streaming the training data looked fine on paper but gave 2 to 3% less throughput than downloading shards locally first, even on a healthy network, and occasional network stalls cost more than that.

An ablation compared the 3.848B model against a 3.128B version with value embeddings disabled, both run to 12,500 steps (29B tokens): with value embeddings, eval loss was 2.1075 and CORE 0.3147; without them, loss was 2.1171 and CORE 0.3047, a 0.46% worse loss and 3.2% lower CORE for 19% fewer parameters, at identical throughput since the embeddings are lookups with almost no extra compute. By the developer's own pricing of that gap (baseline loss fell 0.0194 between steps 10,000 and 12,500, and the value-embedding advantage was 0.0096, about half of that), the embeddings bought the equivalent of roughly 1,200 of the run's 25,000 training steps for 19% more parameters, worth about 5% more training. CORE moved about seven times more than loss did between the two runs, which the developer attributes to CORE being an accuracy metric centered against a random baseline: items near the decision boundary flip on small logit changes, and centering amplifies relative differences while scores are still low.

The write-up's central methodological finding concerns context length. Three of CORE's 22 tasks have prompts that almost never fit inside a 1024-token context: SQuAD (10,570 of 10,570 prompts cropped, 100%), BoolQ (3,265 of 3,270, 99.8%), and a BIG-bench language-identification task (9,965 of 10,000, 99.7%). SQuAD's score did not merely stagnate, it decayed monotonically to exactly zero across the run: 0.1478, then 0.0617, 0.0099, 0.0007, and finally 0.0000. SQuAD's prompts are 10-shot, with a median length of 1,998 tokens that never fits in 1024, and the training harness keeps only the last max_seq_len tokens of an over-length prompt, so it preserved the roughly 169-token test passage and question at the end but discarded the worked examples that taught the expected answer format. Since SQuAD is scored on exact-token match, an early, high-entropy model would occasionally emit a short, generic completion that happened to match, and as the model's language got more fluent those accidental hits disappeared, so it scored steadily worse on SQuAD as it got better at everything else. BoolQ showed a gentler version of the same pattern, peaking at 0.6294 at step 10,000 before declining to 0.5131, while the language-identification task never moved off chance at all.

Rerunning the identical recipe at 2048-token context instead of 1024, halving the micro-batch to hold VRAM constant and stopping early at about 28,000 of a planned 32,000 steps to save the last hours of GPU rental, raised CORE from 0.3384 to 0.3840, a 9% throughput cost (480K to 437K tokens per second); because the run was stopped early, its own author flags the 0.3840 figure as a lower bound rather than a fully converged number. At the matched 20,000-step mark, the two runs had eval losses within 0.0004 of each other (2.0160 versus 2.0164) yet differed by 0.034 on CORE, which the developer calls a surprisingly low correlation between the two metrics. Of the CORE gain, SQuAD and BoolQ alone accounted for 83%: SQuAD rose from 0.0000 to 0.3114 (cropped prompts falling from 100% to 47%) and BoolQ from 0.5131 to 0.7095 (99.8% to 3.2% cropped), with BoolQ's raw gain of +0.196 becoming a centered +0.517 because CORE centers scores against BoolQ's 0.5 random baseline. The other 19 tasks combined moved only +0.008, roughly what the extra 14% of training tokens (65.3B against 57.3B) would be expected to buy on its own; the language-identification task, still the hardest in the whole benchmark for this model, moved only +0.005 despite prompts cropped falling from 99.7% to 14%. Two tasks regressed for reasons left unexplained: commonsense_qa dropped 0.072 and cs_algorithms 0.031. The developer's own conclusion is that the longer context was worth paying for as a measurement decision, letting three otherwise-stuck tasks actually be scored, rather than as a quality improvement in its own right; 1024 tokens remains the cheaper way to reach a good CORE score during training itself.

Four choices were carried over from nanochat without independent testing: the peak learning rate, taken from nanochat's sqrt(768/d_model) scaling rule; the move from cosine to trapezoidal decay, without sweeping other possible schedules; QK-norm, left on by default; and the GQA head ratio. The developer is explicit that this means trusting Karpathy's results transfer to a different model, dataset and scale. Listed open questions for future work include comparing value embeddings against spending the same 721M parameters elsewhere, rerunning the 1024-versus-2048 comparison at matched wall-clock time rather than matched steps, explaining the commonsense_qa regression, sharding the Muon optimizer state the way nanochat's ZeRO-2 implementation does (the current setup uses plain DistributedDataParallel, with every GPU holding a full, redundant copy of the optimizer state), and further data analysis on the CORE benchmark and the ClimbMix dataset themselves. The piece closes by comparing itself to GPT-2: a 2019 result from a well-funded lab with a large team, scoring 0.2565 on CORE, that this solo, evenings-only project beat by a wide margin seven years later for $998 on hourly-rented hardware.

Key facts

  • An independent developer trained a 3.848B-parameter Llama-style model, little-lm, from scratch for $998 (65.3B tokens, 43 hours on 8 rented B200 GPUs), reaching a CORE score of 0.384.
  • At a comparable cost, that beats Andrej Karpathy's nanochat d32 (about 1B parameters, CORE 0.310, roughly $1,000); OpenAI's 2019 GPT-2 (1.5B parameters) scores 0.2565 on the same benchmark.
  • Five changes turned an earlier failed 858M-parameter run (PIQA 60.45%, worse than the seven-times-smaller GPT-2 124M) into the successful recipe: a trapezoidal learning-rate schedule, the Muon optimizer for matrix parameters, the ClimbMix dataset, FP8 training with vocab padding, and a shorter 1024-token context.
  • Three of CORE's 22 tasks have prompts that almost never fit in a 1024-token context; SQuAD's score decayed monotonically to exactly zero because its ten worked examples got truncated while only the short test passage survived.
  • Rerunning at 2048-token context raised CORE from 0.3384 to 0.3840 at a 9% throughput cost, with SQuAD and BoolQ alone responsible for 83% of that gain; a separate ablation found the model's ResFormer-style value embeddings (19% of parameters) worth about 3.2% more CORE.

Why it matters

The headline result, beating a comparably priced result from Andrej Karpathy's well-known nanochat project for $998, is itself the point: a competitive, from-scratch small-model training run has become something one engineer can do on rented hardware in their evenings, not something that needs a lab. The closing comparison makes it explicit: GPT-2's 1.5B model, a 2019 result from a large, well-funded team, scores 0.2565 on the same CORE benchmark, and this project clears that by a wide margin seven years later for under $1,000. Beyond the headline number, the piece is an unusually transparent training report: a published recipe built on hard lessons from a failed 858M-parameter attempt, backed by paired ablations that isolate what each choice was actually worth rather than just asserting that it helped.

Who it affects

Anyone training small or mid-size language models on a limited budget gets a concrete, itemized recipe: which optimizer to use where (Muon for matrix parameters, AdamW elsewhere), which dataset outperformed FineWeb-Edu, and which precision and padding tricks were worth the engineering time. Engineers who use the CORE benchmark specifically should read the context-length findings closely: they show CORE scores can swing heavily for reasons that have nothing to do with model quality, which matters for anyone comparing models scored under different context settings. Individuals or small teams weighing whether to rent GPU time for their own training runs get a fairly literal cost and throughput breakdown to work from, down to the per-optimization gains and dollar totals.

How to use it

There is no product to buy here: little-lm is the author's own training framework, and the post publishes the recipe rather than a service. The appendix gives the full flattened YAML config: a 3072-hidden-size, 28-layer model with 24 query heads and 8 key/value heads, non-gated relu-squared MLPs at a 4x intermediate ratio, QK-norm, a logit softcap of 15.0, and separate per-parameter-group learning rates for matrix weights (Muon), embeddings, the LM head, value embeddings and scalar parameters, most of them derived from nanochat's sqrt(768/d_model) scaling rule. The post does not say whether little-lm's own code, the trained weights, or the ClimbMix data shards are released publicly; the only repository link in the article points to Karpathy's nanochat, not to little-lm itself.

How solid is it

The claims rest on matched-condition comparisons rather than one-off anecdotes: the value-embedding ablation reruns the same config with and without them to the same 12,500-step checkpoint, and the context-length claim reruns the identical recipe at 1024 versus 2048 tokens. GPU-level diagnostics back up the throughput numbers, including 92% SM activity and roughly 1,047 TFLOP/s sustained per B200 (about 25% of Blackwell's dense FP8 peak, 50% of its bf16 peak), consistent with a compute-bound run rather than an idle one. Working against that: the 2048-context run was stopped early, at roughly 28,000 of a planned 32,000 steps, so its own author calls the resulting 0.384 CORE score a lower bound rather than a converged number; several hyperparameters (peak learning rate, QK-norm, the GQA ratio) were carried over from nanochat without independent testing; and every result is a single run of one person's own project, not an average over repeated seeds.

Risks and caveats

The piece's own biggest finding doubles as its clearest warning: at a matched training step, the two context-length runs had eval losses within 0.0004 of each other but a 0.034 gap on CORE, almost entirely from three tasks whose prompts simply did not fit in the shorter context. That means a CORE gain driven by context length can look like a quality gain without being one, worth remembering when comparing any two models on CORE without checking whether every task was actually measurable at the context length used. A couple of results go unexplained: commonsense_qa's score dropped 0.072 and cs_algorithms' dropped 0.031 when context length increased, with no diagnosis offered. And the entire comparison is against a project the author explicitly built on and benchmarked against throughout, using a metric both projects were being tuned to score well on, worth keeping in mind when reading the head-to-head numbers as an unqualified win.

“Work that needed a lab can now be done by a single engineer in the evenings.”

— the post's author, Hugo Vergnes