A 5B Model Frozen at Fifth Grade. Fine-Tuning Couldn't Move It.

TL;DR

A team from the Max Planck Institute for Intelligent Systems, the ELLIS Institute Tübingen, and ETH Zürich built LittleLearner: an 88 billion token pretraining corpus with everything taught above US Grade 5 stripped out, plus 5B, 1.3B and 0.6B models trained on it from scratch alongside matched controls trained on the unfiltered data. The models chat fine and then hit a wall the instant you ask for sixth-grade material. The interesting part is what happened next: the researchers threw scale, SFT plus GRPO post-training, and few-shot in-context learning at that wall, and none of it moved. The paper went up on August 13 and reached the front page of Hacker News today. Weights are on Hugging Face. The corpus is not, yet.

Why anyone built a deliberately stunted model

The honest problem with studying what a frontier model learned is that you have no idea what it read. Every claim about emergent reasoning, memorization, or knowledge injection runs into the same objection: maybe it was just in the training data. When the training data is most of the crawlable web, you cannot rule that out.

Trying to attribute a capability in a web-scale model is like working out which dish gave you food poisoning after eating everything at the buffet. LittleLearner is the researchers building a buffet with three dishes.

The corpus, called LittleCurriculum, is distilled from FineWeb-Edu through five filtering stages aligned to Common Core K-5 standards:

• Age-of-acquisition pre-filter. Drop any document where more than 5% of words are acquired above age 12.
• LLM-as-judge annotation. Label a seed set with a DSPy-optimized judge, then train a classifier on those labels.
• Curriculum classification. A FastText pass for cheap recall, then a ModernBERT pass for precision.
• Symbolic filtering. Regex removal of mathematical notation that leaks past the text classifiers.
• Frequency sampling. Targeted removal of documents carrying beyond-K-5 vocabulary.

The boundary is not a slogan, it is measured

The obvious failure mode for a filtering pipeline like this is that it half works, and you end up studying a model whose supposed boundary is actually a smear. So the team checked it against a ground-truth set built from Common Core text and against WeeBit, an external readability benchmark they did not design the filters around.

Roughly 35% of genuinely K-5 material survives, which is wasteful but harmless. Zero percent of the ground-truth beyond-K-5 material survives. On WeeBit, 2.48% of out-of-scope documents slipped through, and manual inspection of those found actual out-of-scope concepts in only 0.05%. The wall is where they say it is.

The models behave accordingly. On Jeopardy-style science questions, LittleLearner tracks the unfiltered control until the exposure horizon and then falls off a cliff, while the control stays flat. On MathCAMPS, which turns 44 Common Core standards from K through 8 into thousands of graded word problems, LittleLearner declines disproportionately at higher grade levels against both the unfiltered control and a Gemma 2B baseline.

Three attempts to teach it sixth grade

This is the part that should interest anyone who has ever promised a stakeholder that fine-tuning will teach the model their domain.

Scale

Going from 0.6B to 5B parameters improves mathematical reasoning inside the training exposure, exactly as you would expect. Grade 8 MathCAMPS performance stays at floor across all three sizes. More parameters let the model do more with what it read; they do not conjure what it did not read.

Post-training

SFT followed by GRPO lifts both LittleLearner and the unfiltered control above their base performance within K-5. Beyond K-5, post-training only modestly improves LittleLearner, and here is the detail that does the real damage: there was no difference between post-training on K-5 data and post-training on beyond-K-5 data. The out-of-scope curriculum they fed it might as well have been a phone book. Whatever the gain was, it came from teaching the model to use what it already had, not from the new material.

In-context learning

The team synthesized three problems per Common Core standard with hand-authored chain-of-thought and put them in the prompt. Inside K-5, modest gains. Beyond K-5, no gain at all. Adding worked explanations on top of the examples changed nothing.

What this actually licenses you to conclude

The clean version: for this model family, at this scale, post-training and few-shot prompting are levers over knowledge that pretraining already installed. They are retrieval and formatting, not installation. If your plan is to fine-tune a domain the base model has genuinely never seen, this paper is a controlled counterexample to the thing everyone assumes.

Now the caveats, and they matter:

• They tested SFT plus GRPO and ICL, not large-scale continued pretraining. Nobody here showed that pouring another 50B tokens of eighth-grade math into the base model fails. That experiment is not in the paper.
• The context window is 4096 tokens. Three worked examples per standard is not the same as stuffing a retrieval corpus into a 200K window, so read this as a limit on few-shot prompting, not a verdict on RAG.
• 5B is small, and the domain is elementary school. The boundary is unusually sharp because the filter was unusually aggressive. Real domain gaps in real base models are fuzzier.

What actually shipped

Fifteen model repos are live on Hugging Face: littlelearner-{0.6b,1.3b,5b}-base, matched unfiltered-*-base controls, and GRPO math-expert variants of both arms. The matched controls are the part worth stealing; a filtered model without its unfiltered twin proves nothing.

The 5B base card lists a Qwen3-style dense transformer at 5.04B parameters, hidden size 3072, 44 layers, 24 query and 8 KV heads, FFN 9216, 4096 context, and a custom 64k byte-level BPE tokenizer with per-digit splitting and ChatML tokens. In-domain bits-per-byte is 0.536. Training used a WSD schedule, sharded Muon, MXFP8, and Megatron-Core on eight B200s. The license field reads other, so read the terms before you build anything commercial on it.

One gap: the abstract says the team releases both LittleCurriculum and LittleLearner. As of writing, the datasets tab on the Hugging Face org says "None public yet." The 88B-token corpus is the artifact everyone actually wants, and it is the one still behind the curtain.

Key Takeaways

• 88B tokens, nothing above Grade 5. LittleCurriculum keeps roughly 35% of in-scope material and 0% of ground-truth out-of-scope material, so the capability boundary is measured rather than asserted.
• Scale does not extend the boundary. Grade 8 MathCAMPS performance sits at floor across 0.6B, 1.3B and 5B.
• Post-training on out-of-scope data performed no better than post-training on in-scope data. That is the single most quotable result in the paper.
• Few-shot prompting beyond the boundary produced no gain at all, with or without hand-written chain-of-thought.
• Weights are out, the corpus is not. Fifteen repos including matched unfiltered controls, license listed as "other," datasets tab still empty.
• Scope your conclusions. This is a 5B model with a 4096 context tested under SFT, GRPO and few-shot ICL. It is not a result about continued pretraining or long-context retrieval.

Sources: LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure (arXiv 2608.13545), full paper HTML, LittleLearner project page, Hugging Face org, littlelearner-5b-base model card, FineWeb-Edu dataset, MathCAMPS, Hacker News discussion