This post from zhichai.net introduces and explains the paper "Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views" by Joseph Lee, Yidi Huang, and Dokyoon Kim (arXiv:2609.04168).
Core idea
The author opens with an analogy: Student A reads a textbook ten times verbatim, while Student B reads it once and then studies tutoring books, videos, and classmates' notes—the same concept from five angles. The paper's findings suggest Student B's strategy wins: during LLM pre-training, allocating part of the token budget to auxiliary views (diverse expressions of the same knowledge) outperforms repeating the original text.
Background: how LLMs acquire knowledge
Pre-training is superficially simple—predict the next token—but the model builds a knowledge network underneath. Prior work shows repetition matters: facts appearing 1000 times are learned far better than facts appearing once, which is why common knowledge is well captured while obscure facts often fail. The paper asks a finer question: if repetition is necessary, repeat what? Mechanical repetition of the same sentence, or different expressions of the same knowledge?
Experimental setup
The authors use a teacher–student framework: a stronger teacher model generates multiple "views" of a fact (e.g., "water boils at 100°C" presented as an explanation, a contrast, an application, and a historical note), and a smaller student model is trained on them to measure knowledge acquisition.
Five key findings
1. Repetition is necessary, but blind repetition is inefficient. A fact seen once is rarely learned regardless of view diversity. But under an equal token budget, splitting the budget between repetition and auxiliary views beats pure repetition.
2. Paraphrasing helps only under certain conditions. Simple paraphrases help with small batch sizes but add little with large batches—suggesting paraphrase diversity interacts with gradient-update noise, which large batches already smooth out.
3. Effectiveness does not depend on teacher model strength. Even a weaker teacher's auxiliary views help, as long as they offer different angles—like a mid-level TA explaining a concept three ways possibly outperforming a top professor's single explanation.
4. Two knowledge types: contextual vs. foundational. Auxiliary views work best when the model already has a basic framework. Learning on top of existing knowledge consolidates and extends it; for wholly unfamiliar domains, views help less. This explains why fine-tuning a foundation model is more efficient than training from scratch.
5. Mechanism: layer-wise biases and compression. Auxiliary views calibrate layer-wise biases, stabilizing knowledge representations, and lead to more compressed encoding—the model represents knowledge more efficiently, analogous to a human holding a tighter, more robust mental model after understanding a concept from multiple angles.
Why diversity drives emergent abilities
The paper offers a causal account of why diverse pre-training data improves generalization: encountering varied expressions of the same knowledge forces the model to extract invariants—the shared core structure—i.e., to learn abstract representations. The author speculates that emergent abilities with scale may stem from larger models having the capacity to encode statistical regularities across more views simultaneously.
Practical implications for LLM training
- Data curation > data volume. A carefully chosen, diverse 1T corpus may beat 10T of repetitive web text.
- Synthetic data potential. Since view quality doesn't require a top teacher, mid-tier models can generate multi-view training data to improve small models.
- Curriculum learning. Teach foundations first, then introduce diverse perspectives—mirroring human education.
- Multilingual advantage. The same fact expressed in different languages naturally constitutes auxiliary views, potentially explaining why multilingual models generalize better.
Conclusion
The author closes: repetition is necessary, but what you repeat matters more than how many times. When training the next trillion-parameter model, the better question is not "how much data do we have?" but "how many ways of seeing the world does our data provide?" Quoting Feynman: "If you can't explain something in simple terms, you don't truly understand it."
References cited in the post
1. Lee, J., Huang, Y., & Kim, D. (2026). *Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views*. arXiv:2609.04168. 2. Chang, H., et al. (2024). How Do Large Language Models Acquire Factual Knowledge During Pretraining? arXiv:2406.11813. 3. Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. 4. Cheng, D., et al. (2024). Instruction pre-training: Language models are supervised multitask learners. EMNLP 2024. 5. Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI Technical Report.