Why Reading a Textbook Three Times Beats Reading It Ten: LLM Pre-training Learns Better With Auxiliary Views

Forum topic · 小凯 · 2026-09-05

Summary

A Chinese tech forum post explains a recent arXiv paper (2609.04168) showing that large language models acquire knowledge more efficiently during pre-training when the same fact is presented through multiple auxiliary views rather than repeated verbatim. Using a teacher-student experimental framework, the authors found that: repetition is necessary but should be partially traded for diverse paraphrases and perspectives under a fixed token budget; simple paraphrasing helps mainly with small batch sizes; auxiliary views remain effective even when generated by weak teacher models; benefits are strongest when the model already has foundational knowledge; and the mechanism works through layer-wise bias calibration and compression. The post argues this explains why data diversity drives generalization and possibly emergent abilities, and draws practical lessons for data curation, synthetic data, curriculum learning, and multilingual pre-training.

Paper: Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views Authors: Joseph Lee, Yidi Huang, Dokyoon Kim arXiv: 2609.04168

This forum post explains the paper with a study-strategy analogy: Student A reads a textbook ten times verbatim, while Student B reads it once and then studies the same concepts from five different angles (guidebooks, videos, classmates' notes). Research on LLM pre-training suggests Student B's approach — learning via auxiliary views — works better.

Background: How LLMs Acquire Knowledge

Pre-training is nominally just next-token prediction, but behind it the model builds a knowledge network. It is well known that repetition matters: a fact appearing 1,000 times in training data is far more likely to be memorized than one appearing once — which is why common facts ("Paris is the capital of France") are learned well while obscure ones are not. The paper asks a finer question: if repetition is necessary, what should be repeated? The same sentence mechanically, or the same knowledge expressed differently?

The Controlled Experiment

The authors use a teacher-student framework:

1. A stronger teacher LLM generates different "views" of a piece of knowledge (e.g., "water boils at 100°C" rendered as an explanation, a comparison with high-altitude boiling, an application like pressure cookers, and a historical note). 2. These views train a smaller student LLM. 3. The student's knowledge acquisition is measured.

The core comparison: does training on four different views beat repeating the original sentence four times?

Five Key Findings

1. Repetition is necessary, but blind repetition is inefficient. A fact seen only once is rarely acquired regardless of view diversity. But under the same token budget, allocating part of the repetition budget to auxiliary views beats repeating the original text only.

2. Paraphrasing helps only under certain conditions. Simple paraphrases help with small batch sizes but add little benefit with large batches. The authors suggest this relates to gradient-update noise: small batches are noisy, so paraphrase diversity stabilizes learning; large batches already smooth the noise.

3. Auxiliary views don't depend on teacher strength. Even views generated by a weak teacher help, as long as they offer different angles — like a mid-level TA teaching a concept three ways outperforming a top professor teaching it one way.

4. Two forms of knowledge: contextual and foundational. Auxiliary views work best when the model already has a foundational framework in place. For completely unfamiliar domains, their benefit is limited — consistent with fine-tuning after pre-training being more efficient than training from scratch.

5. Mechanism: layer-wise biases and compression. Auxiliary views appear to calibrate layer-wise biases and lead to more compressed, efficient knowledge representations — the model encodes the same information with fewer effective parameters.

Why Diversity Enables Emergence

The paper offers a causal account of why data diversity improves generalization: seeing multiple expressions of the same knowledge forces the model to extract invariants — the shared core structure — i.e., to learn abstract representations. This may also partially explain emergent abilities at scale: larger models have capacity to encode statistical regularities across more views simultaneously, extracting deeper abstractions.

Practical Implications for LLM Training

Data curation > data volume: a curated, diverse 1T corpus may beat a repetitive 10T one.
Synthetic data: since view quality doesn't require a top-tier teacher, mid-capability models can cheaply generate multi-view training data for smaller models.
Curriculum learning: build foundational concepts first, then reinforce with diverse views — mirroring human education.
Multilingual training: the same fact expressed in different languages is naturally an auxiliary view, which may partly explain why multilingual models generalize better.

Conclusion

The paper does not say repetition is unimportant — it confirms repetition is necessary. Its core point: what you repeat matters more than how many times you repeat it. As the post puts it, echoing Feynman: "If you can't explain something simply, you don't really understand it" — auxiliary views essentially force the model to re-express knowledge in its own words.

References

1. Lee, J., Huang, Y., & Kim, D. (2026). *Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views*. arXiv:2609.04168. 2. Chang, H., et al. (2024). How Do Large Language Models Acquire Factual Knowledge During Pretraining? arXiv:2406.11813. 3. Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. 4. Cheng, D., et al. (2024). Instruction pre-training: Language models are supervised multitask learners. EMNLP 2024. 5. Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI Technical Report.