He Trained a 3.8B Model for $998. One Task Decayed to Zero.

TL;DR

On September 4, Hugo Vergnes published little-lm, a full write-up of pretraining a 3.848B-parameter model from random weights on eight rented B200s: 65.3B tokens, 43 hours, $998, and a DCLM CORE score of 0.384. GPT-2 scores 0.256525 on the same metric. The headline number is fun. The part worth your time is the forensics: three of CORE's 22 tasks never fit inside his training context, one of them decayed monotonically to exactly zero the longer he trained, and 83 percent of his final score bump came from fixing the harness rather than the model.

---

What $998 buys in September 2026

The reference point is Andrej Karpathy's nanochat, which exists to make sub-$1,000 pretraining legible. Its $1,000 tier run reports the receipts in full: 1,879,048,192 parameters, 37.58B tokens, 30.8 hours on an 8x H100 node billed at $24/hour, CORE 0.3168. Karpathy himself put the real spend "closer to $800."

little-lm is roughly twice that size on twice the tokens, for a bit more money, on newer silicon.

Divide the bill by the clock and you get about $23 an hour for the node, roughly $2.90 per B200-hour. That is the actual headline for anyone deciding whether to rent: a frontier-class accelerator now costs about the same per hour as a mid-range one did two years ago, and the recipe to use it well is public.

The five changes that turned a failure into a result

Vergnes did not start here. His first serious attempt was an 858M model on FineWeb-Edu, 16.4B tokens, 5.8 days on a single A100. It scored 60.45% on PIQA. GPT-2's 124M variant scores about 63%. Six days of compute to lose to a model seven times smaller from 2019 is the kind of result that either ends a project or produces a good post-mortem.

Five changes came out of it:

• Trapezoidal learning rate. Warm up 5%, hold flat, linear cooldown over the final 50% to 5% of peak. His cosine-to-zero curve went flat at roughly 70% of steps, so the last 30% of the budget bought nothing. In the 3.8B run, eval loss was still falling at the final step.
• Muon for matrix parameters, AdamW for the rest. Newton-Schulz orthogonalization costs about 25% per optimizer step in a shallow-accumulation benchmark, but at 7 gradient-accumulation steps that dilutes to roughly 4%.
• ClimbMix instead of FineWeb-Edu. Nvidia's 400B-token filtered mix, and by his account the single largest jump in convergence speed.
• FP8 plus vocab padding. All three GEMMs in FP8 with dynamic tensorwise scaling, and the vocabulary padded from 50,257 to 50,304 so it lands on a multiple of 64. Together, +33% throughput.
• 1,024 context instead of 2,048. Halve the context, double the batch at fixed memory. This is the one that came back to bite him.

The throughput work happened on a gaming card

Before renting anything, he tuned on a single RTX 5090 and took an 858M config from 26,144 to 37,621 tokens/sec. The most interesting win is the least intuitive: Liger's fused linear cross-entropy is 6% slower head-to-head at matched batch size, because it never materializes the full logits tensor. It also frees 8GB of VRAM, which buys a bigger micro-batch, which more than pays the 6% back. He notes Claude rejected the change on the raw benchmark number, which is a fair summary of what happens when you optimize a single metric in isolation.

Keeping optimizer master weights in bf16 rather than fp32 was the other big one: VRAM down 27%, throughput from 640K to 1.4M tok/s on a 1.5B config, a 2.2x jump, for CORE 0.22 versus 0.23 at 4,000 steps. On the rented node he sustained about 1,047 TFLOP/s per B200, 92% SM activity, roughly 25% MFU against Blackwell's dense FP8 peak.

Then the benchmark started measuring the harness

Here is the part that generalizes past one hobby run. CORE aggregates 22 tasks, and each contributes a centered score: accuracy minus the random baseline, divided by one minus that baseline. Three of the 22 have prompts that essentially never fit in 1,024 tokens.

• SQuAD: 10,570 of 10,570 prompts cropped (100%)
• BoolQ: 3,265 of 3,270 cropped (99.8%)
• BIG-bench language identification: 9,965 of 10,000 cropped (99.7%)

SQuAD in the DCLM bundle is a 10-shot task, median prompt length 1,998 tokens on his eval data. His harness truncates by keeping the last max_seq_len tokens. The test passage sits at the end and is only about 169 tokens, so it always survived. What got cut, every single time, were the ten worked examples that teach the model the expected answer format. Picture a student handed the exam question and the reading passage with the front page of worked examples torn off, then graded on exact string match. That is the whole bug.

The score got worse as the model got better

SQuAD did not stagnate. It fell: 0.1478, then 0.0617, 0.0099, 0.0007, and finally 0.0000. Models do not normally get worse at a task the longer they train, which is exactly why this is worth flagging. His explanation is convincing: an early, high-entropy model occasionally emits something short and generic that happens to exact-match the gold answer. As it sharpens, it commits to fluent, well-formed continuations, and the accidental hits stop happening. Learning English cost it every lucky guess it had.

BoolQ shows a gentler version of the same curve, peaking at 0.6294 around step 10,000 and sliding to 0.5131. Language identification never leaves chance at all.

Re-running at 2,048 context

Same recipe, same tokens per optimizer step, context doubled and micro-batch halved. CORE went from 0.3384 to 0.3840, and he stopped that run early enough that the cooldown never finished, so 0.3840 is a lower bound.

The cleanest comparison in the whole write-up is at step 20,000, where the two runs sit at eval loss 2.0160 and 2.0164, identical to four decimals, and 0.034 apart on CORE. Same model quality by loss. Different score by harness.

He is blunt about what that means: 2,048 was worth paying for as a measurement decision, not a quality one. It cost 9% throughput (480K to 437K tok/s) and, outside the three tasks that could not be scored at 1,024, bought almost nothing. If you are training small and reporting CORE, 1,024 is a cheap way to look worse than you are.

The one ablation he actually ran

Value embeddings were 721.2M parameters, 19% of the model, spread across 14 tables on every other layer. He trained the identical config with them off, out to 12,500 steps and 29B tokens:

• Loss 2.1075 with, 2.1171 without: 0.46% better
• CORE 0.3147 with, 0.3047 without: 3.2% better
• Throughput 479,445 vs 477,908 tok/s: identical, because lookups cost memory, not FLOPs

Then he prices it, which is the move most ablations skip. Between steps 10,000 and 12,500 his baseline loss fell 0.0194. The value-embedding advantage is 0.0096, about half of that, so roughly 1,200 steps out of 25,000. Nineteen percent more parameters bought about five percent more training.

The second finding there is a warning label for anyone tuning against CORE: the metric moved seven times more than loss did, because it is an accuracy metric centered against a random baseline, so items near the decision boundary flip on tiny logit changes and relative differences get amplified while scores are still low.

The caveats, stated plainly

• There is no repository. little-lm is described in detail but not published; there is no public code or weights to clone as of today. You get the recipe and the numbers, not the artifact. nanochat, which he built on top of conceptually, is public.
• ClimbMix is CC BY-NC 4.0. Non-commercial. Whatever you train on it inherits that conversation.
• Four knobs were never ablated: peak learning rate (inherited from nanochat's sqrt(768/d_model) heuristic), the trapezoidal schedule itself, QK-norm, and the GQA ratio. He says so directly, and calls it a defensible way to spend a small budget: someone else already paid for those experiments.
• One unexplained regression. commonsense_qa dropped 0.072 at the longer context, and cs_algorithms 0.031, with no mechanism offered.

Key Takeaways

• A 3.848B model trained from scratch to 0.384 CORE for $998 in 43 hours on 8x B200, versus GPT-2's 0.256525 and nanochat d32's 0.3168 at roughly $800.
• Check your eval harness before you check your model: three of CORE's 22 tasks were 99.7% or more truncated at 1,024 context, and fixing that was worth 0.034 CORE at matched eval loss.
• A truncating harness that keeps the tail of a prompt will silently delete your few-shot demonstrations while leaving the question intact, which looks like a model failure and is not.
• A benchmark score can fall while a model improves. SQuAD went 0.1478 to exactly 0.0000 because exact-match rewards lucky short outputs that a sharper model stops producing.
• Slower-per-step kernels can be faster per dollar. Liger's fused cross-entropy lost 6% head-to-head and won 44% cumulatively once the freed VRAM went into batch size.
• Value embeddings at 19% of parameters bought about 5% more effective training and cost nothing in throughput, which is a good trade at this scale and an untested one against spending those parameters elsewhere.

---

Sources: Hugo Vergnes, "Training a 3.8B LLM to 0.384 CORE for $998", karpathy/nanochat, nanochat $1000 tier run report, DataComp-LM (DCLM) paper, Nvidia ClimbMix dataset card.