A model that 'doesn't know Schrödinger's cat'
Ask a freshly trained 5B-parameter language model: "What is Schrödinger's cat?"
It answers, in earnest: "Schrödinger's cat is a cat with two faces."
This isn't a hallucination — it's genuine ignorance. The model isn't spouting nonsense; it's assembling the most plausible answer from the vocabulary it has. It knows what a "cat" is, doesn't know what "Schrödinger" is, so it guesses.
The same model, asked "What is gravity?", answers: "Gravity is the force that pulls everything down." — a standard elementary-school answer.
The model is called LittleLearner: a 5B-parameter language model deliberately "locked" inside the scope of the US K-5 curriculum during training. It's not underpowered, not architecturally outdated, not undertrained — it has simply never seen any content beyond sixth grade.
The paper's authors did something unprecedented: instead of controlling novelty at evaluation time, they controlled the knowledge boundary at training time. Using a sophisticated multi-stage filtering pipeline, they kept only content matching K-5 curriculum standards from FineWeb-Edu's massive web text, built an 88B-token "elementary curriculum corpus" (LittleCurriculum), and trained a 5B model from scratch.
The result is a controlled sandbox — researchers finally know exactly what the model "has and hasn't seen," enabling a question that is fundamentally unanswerable with ordinary LLMs:
> What does pretraining actually determine? Can post-training break the boundary drawn by pretraining?
Why an "elementary school classroom"
Researchers of LLMs face a long-standing embarrassment: to know whether a capability was "learned" or "already seen," you'd need to know exactly what's in the pretraining data. You don't.
Data contamination inflates benchmark scores. The reversal curse (learning "A is B" without "B is A") exposes fragile generalization. RLHF, chain-of-thought, and few-shot learning improve performance — but is this genuinely new capability, or just eliciting knowledge already present in pretraining?
Previous solutions worked on the evaluation side: ever-harder benchmarks (GPQA, Humanity's Last Exam, MATH-B) that models presumably haven't seen. But that's indirect control — you're betting the question is new enough.
LittleLearner inverts the approach: control the training, not the evaluation.
Lock the model in an "elementary classroom" where it only sees K-5 content. Now you know it has never seen quadratic equations, quantum mechanics, or calculus notation. Any performance on Beyond-K-5 tasks must be true generalization or true reasoning — not replayed memory.
It's like studying child cognitive development with children of "controlled exposure" — you know which books they've read — rather than grabbing strangers off the street and guessing what they've learned.
Six filtering layers: how to extract "only K-5 content"
Filtering "K-5 content" from web-scale text is harder than it sounds. The paper's six-stage pipeline follows one principle: precision over recall — it's better to miss some K-5 content than to let Beyond-K-5 content leak in.
1. Age-of-Acquisition pre-filtering. A psycholinguistic database records the age at which humans typically learn each English word. If more than 5% of a document's words are learned after age 12, drop it. Words missing from the database (10%) are imputed via word-frequency regression — low-frequency words tend to be learned later. 2. LLM-as-Judge labeling + classifier training. Gemini Flash labels a sample of documents with grade levels (full annotation would cost $46M), then two classifiers are trained: lightweight FastText for a first pass, and the more expensive but accurate ModernBERT for a refined pass. 3. Symbol filtering. Regex matching for math symbols — ∑, ∫, ∂, quadratic expressions — triggers an immediate discard. This removes only 0.1% of documents but precisely cuts off the tail of mathematical formulas. 4. Frequency sampling. Words that appear far more often in Beyond-K-5 text than in K-5 text are identified, and documents containing them are removed. Since models can learn facts from sparse occurrences, better safe than sorry. 5. Validation on CommonCoreText (CCSS-aligned textbooks): Beyond-K-5 retention near 0%, K-5 retention ~35–42%. 6. External validation on WeeBit (an independent graded corpus), confirming the same boundary quality.
The pipeline is an engineering achievement in itself. It isn't "selecting good content" — it's "drawing a hard boundary." Noise inside the boundary is fine; leakage outside is not.
Three interventions, three failures
With LittleLearner trained, the authors ran three experiments asking the same question: can the model be pushed beyond the K-5 boundary?
Experiment 1: Scaling
Scaling LittleLearner from 0.6B → 1.3B → 5B: within K-5, scale helps clearly; on Beyond-K-5, scale does nothing. The 5B model performs just as poorly on eighth-grade math as the 0.6B model.
But there's a fascinating side finding: the 0.6B LittleLearner beats the 0.6B Unfiltered model on K-5 math. Why? The unfiltered 0.6B model's capacity is too small — it gets "distracted" by high-school algebra and calculus content and never masters elementary arithmetic. LittleLearner's 0.6B model, having only seen K-5, devotes all its capacity to elementary content — focus beats erudition.
This "focus advantage" vanishes at larger scale — a 5B model has enough capacity to learn everything well. But in small models, curriculum design is capacity allocation.
Experiment 2: RL post-training (SFT + GRPO)
A subtle design: two post-training variants — one using only K-5 data, one using Beyond-K-5 data. If RL could genuinely "teach new capabilities," the Beyond-K-5-post-trained model should improve on Beyond-K-5 tasks.
Result: neither variant produced significant Beyond-K-5 gains. Even GRPO on Beyond-K-5 data fails — the model lacks the pretraining foundation needed to make sense of that data.
This directly addresses a hot debate: are RL gains "newly learned" or "elicited" from pretraining? LittleLearner's answer: elicited. When pretraining contains nothing relevant, RL cannot conjure capability from nothing.
Experiment 3: In-context learning (ICL)
Giving the model few-shot Beyond-K-5 worked examples: slight gains within K-5, completely ineffective beyond. Output format shifts toward the examples (answers get shorter), but reasoning does not improve.
Even more interesting: textual explanations (instead of examples) have zero effect — the answers don't even change format. The model isn't "failing to understand the examples"; it lacks the underlying representations needed to execute the reasoning.
Pretraining is destiny
All three experiments point to one conclusion:
> The pretraining distribution sets a capability ceiling. Scale, RL, and ICL — the three standard interventions — can only amplify capabilities beneath that ceiling; they cannot raise it.
This isn't a "negative result" but an extremely valuable foundational finding:
1. All current "post-training creates capability" narratives need re-examination. If a model improves after RL, it likely "elicited" pre-existing capability rather than "learning" something new. LittleLearner provides a clean environment to distinguish the two. 2. Data determinism gets its strictest validation. Previously "data matters" couldn't rule out architecture/scale/training-trick explanations. LittleLearner controls everything except the training-data scope — and the capability boundary tracks the data boundary precisely. 3. "Emergence" becomes falsifiable. If a model "suddenly" acquires a skill, it may simply reflect relevant pretraining content rather than emergence.
Models don't learn math like children
An unexpected finding: LittleLearner's capability structure doesn't follow human curriculum order. In human curricula, single-digit division precedes multi-digit division. But LittleLearner performs *better* on multi-digit division than single-digit division.
Model learning is statistically driven: multi-digit division text may simply be more frequent, or its patterns easier for transformer attention to capture. This has implications for education science: we cannot assume AI learning paths resemble human ones. Human curricula are valid as a knowledge boundary (known and verifiable), but human learning theory can't predict model capability structure.
The 0.6B small-model paradox
The most captivating finding: a 600M-parameter model that has only seen elementary content beats a 600M-parameter model that has seen everything — on elementary math. With limited capacity, erudition is distraction. In resource-constrained settings (edge models, embedded devices), curriculum design is an invisible capacity amplifier.
A clean experimental arena
LittleLearner's value isn't being "a better model" — it clearly isn't. Its value is being a clean experimental arena. Future directions the paper suggests:
- RL's true discovery capability: run RL on Beyond-K-5 math; if LittleLearner genuinely learns quadratic equations, that's real "machine discovery," not pretraining replay.
- Continual learning: teach a trained LittleLearner new content (e.g., negative numbers) and observe representation change and interference with old knowledge.
- Human-machine comparison: teach the same textbooks to models and children; compare speed and error patterns.
Closing thoughts: pretraining as "constitutional law"
A political-science analogy: pretraining is the constitution, post-training is statute law, ICL is an executive order. The constitution defines what is possible; statutes refine rules within it; executive orders execute within statutes. You can't break the constitution with an executive order — you need an amendment.
For AI engineers: if you want a model to have genuinely new capability in some domain — not prettier formatting, but solving problems it previously couldn't — you must go back to pretraining and check whether the data contains relevant content. RL and prompt engineering dance within the grid that pretraining has drawn. LittleLearner gives this intuition experimental evidence.
To be fair
The paper has a quality I deeply appreciate: honesty. The authors don't overclaim — they note that 5B may be too small to show certain emergent behaviors (ICL may work better at larger scale), acknowledge ~0.05% Beyond-K-5 leakage, and frame the "three failed interventions" as a starting point rather than a negative result.
"Schrödinger's cat is a cat with two faces" is not a clever answer, but it is an honest one. The model doesn't know — and knows it doesn't know. In an AI era full of hallucination and overconfidence, that honesty is itself a contribution.
---
Paper: LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure arXiv: https://arxiv.org/abs/2608.13545 HTML version: https://arxiv.org/html/2608.13545 Authors: Li Fanfei, Zeller Jana, Prada-Corral Manuel, et al. Institutions: University of Tübingen, Max Planck Institute for Intelligent Systems Model scale: 5B parameters (Qwen3 architecture) Training data: LittleCurriculum, 88B tokens, K-5 curriculum-filtered Training resources: 8× NVIDIA B200 GPUs, 100 hours