LittleLearner: Language Models Trained Under Pedagogically Controlled Knowledge Boundaries

Forum topic · 小凯 · 2026-08-15

Summary

Researchers introduce LITTLECURRICULUM, a curated 88B-token pretraining corpus based on U.S. elementary school material that explicitly excludes concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter language model from scratch on this corpus yields LITTLELEARNER, a model with sufficient language competence for open-ended evaluation while having clear, interpretable knowledge and capability boundaries mapped to curriculum guidelines. The corpus and model are released as a developmentally restricted sandbox for studying how language models acquire, represent, and use knowledge within a well-defined training scope. Initial experiments on injecting new knowledge via post-training and in-context learning show these methods help LITTLELEARNER better utilize existing knowledge but do not extend capabilities beyond the curriculum boundary. The work addresses the difficulty of characterizing prior exposure in web-scale training corpora, offering a controlled testbed for research on knowledge acquisition in language models. (arXiv: 2608.13545)

LittleLearner: Language Models Under Pedagogically Controlled Knowledge Boundaries

Field: NLP Authors: Fanfei Li, Jana Zeller, Manuel Prada-Corral, Thaddäus Wiedemer, Prasanna Mayilvahanan, Ryan Cotterell, Wieland Brendel Published: 2026-08-13 arXiv: 2608.13545

Abstract

Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize.

To address this challenge, the authors introduce LITTLECURRICULUM, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5.

Training a 5B-parameter LLM from scratch on LITTLECURRICULUM yields LITTLELEARNER, a model with:

Sufficient language competence for open-ended evaluation
Clear knowledge and capability boundaries mapped to interpretable curriculum guidelines

LITTLECURRICULUM and LITTLELEARNER are released as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope.

First Experiments

The sandbox's utility is illustrated in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LITTLELEARNER better utilize existing knowledge, but do not improve capabilities beyond the curriculum's scope.

Significance

These findings underscore the value of this controlled environment for future investigations into how language models acquire, represent, and use knowledge under well-defined training boundaries.

--- *Auto-collected on 2026-08-15*