Paper: Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views Authors: Joseph Lee, Yidi Huang, Dokyoon Kim arXiv: 2609.04168
---
Opening: Two Study Strategies
Imagine two students preparing for a final exam.
Student A reads the textbook ten times, word for word, believing repetition alone will imprint knowledge.
Student B reads it once, then seeks out study guides, lecture videos, classmates' notes, and documentaries—understanding each concept from five different angles.
Conventional intuition favors A's repetition. But new research shows that B's strategy—learning with "auxiliary views"—offers surprising advantages in LLM pre-training. This explains why data diversity can matter more than sheer volume, and reveals a deeper principle of knowledge acquisition.
---
How Do LLMs "Learn" Knowledge?
On the surface, pre-training is simple: predict the next token. But behind this task, the model builds a vast knowledge network ("cats are animals," "cats like warm places," etc.).
Researchers have long known that repetition matters: a fact appearing 1000 times is memorized far more reliably than one appearing once. This is why common facts ("Paris is the capital of France") are learned well while obscure ones often fail.
The paper asks a finer question: if repetition is necessary, repeat what? The same sentence mechanically, or the same knowledge expressed differently?
---
A Controlled Experiment: Isolating Auxiliary Views
The researchers used a teacher-student framework:
1. A teacher model (a stronger LLM) generates different "views" of a piece of knowledge 2. These views train a student model (a smaller LLM) 3. The student's mastery is tested
Example—the original fact: *Water boils at 100°C.*
- Explanatory view: molecules gain enough kinetic energy to escape the liquid at this threshold
- Comparative view: water boils at lower temperatures at high altitude, showing boiling point depends on pressure
- Applied view: pressure cookers raise the boiling point, cooking food faster
- Historical view: before the 18th century, ignorance of this relation caused errors in altitude measurement
---
Five Counterintuitive Findings
Finding 1: Repetition is necessary, but blind repetition is inefficient. A fact seen once is rarely learned regardless of view diversity. But under an equal token budget, allocating part of the budget to auxiliary views beats spending it all on repeating the original.
Finding 2: Paraphrasing helps only under certain conditions. Simple paraphrases (synonym swaps, restructured sentences) help with small batch sizes but yield almost no extra benefit at large batch sizes. This suggests paraphrasing works via gradient-update noise: small batches are noisy, so diversity stabilizes learning; large batches already smooth the noise.
Finding 3: Effectiveness is independent of teacher strength. Auxiliary views remain effective even when generated by a weaker teacher. An imperfect teacher offering multiple angles can help more than a top expert offering one.
Finding 4: Contextual vs. foundational knowledge. Auxiliary views work best when the model already has foundational knowledge to build on. For completely unfamiliar domains, their help is limited—explaining why fine-tuning pre-trained models is more efficient than training from scratch.
Finding 5: Mechanisms—layer-wise biases and compression. Auxiliary views seem to calibrate layer-wise biases and promote compression: models encode the same information more efficiently, yielding more compact, robust representations.
---
Why Diversity Yields Emergent Abilities
Previously, researchers observed that diverse data improves generalization, but lacked a causal account. This paper proposes auxiliary views as the mechanism: exposure to multiple expressions forces the model to extract invariants—the shared core structure—i.e., to learn abstract representations.
This may also shed light on emergent abilities: perhaps scale matters not because "bigger is better" per se, but because larger models can simultaneously encode statistical regularities from more views, extracting deeper abstractions.
---
Practical Implications for AI Training
1. Data curation > data volume: carefully selected, diverse data may matter more than scale. 2. Synthetic data: since teacher strength matters little, mid-capability models generating multi-view data could boost smaller models. 3. Curriculum learning: teach foundations first, then diversify with auxiliary views—mirroring human education. 4. Multilingual training: the same fact in different languages is naturally an auxiliary view, possibly explaining multilingual models' stronger generalization.
---
Conclusion: From Statistical Parrots to Real Understanding
Student A represents the traditional "big data + big model" approach—effective but inefficient. Student B embodies the new insight: intelligence is not just memorization, but abstracting shared structure from multiple angles.
The paper does not say repetition is unimportant—it confirms repetition is necessary. Its core contribution: what you repeat matters more than how many times you repeat it.
As Feynman said:
> "If you can't explain something in simple terms, you don't truly understand it."
Auxiliary views force a model to re-express knowledge in its own language—the path to genuine understanding.
---
References
1. Lee, J., Huang, Y., & Kim, D. (2026). *Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views*. arXiv:2609.04168. 2. Chang, H., et al. (2024). How Do Large Language Models Acquire Factual Knowledge During Pretraining? *arXiv:2406.11813*. 3. Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. *arXiv:2203.15556*. 4. Cheng, D., et al. (2024). Instruction pre-training: Language models are supervised multitask learners. *EMNLP 2024*. 5. Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. *OpenAI Technical Report*.
---
*Posted 2026-09-06*