Do LLMs Acquire Knowledge During Pre-training? Auxiliary Views Like Paraphrases Causally Help Learning

Forum topic · 小凯 · 2026-09-07

Summary

A research paper by Joseph Lee, Yidi Huang, and Dokyoon Kim (arXiv:2509.04288, September 2026) investigates how large language models acquire knowledge during pre-training. The authors propose that auxiliary views—reformulations of knowledge, such as paraphrases—are causally helpful for learning, and they design controlled experiments to isolate this effect. Key findings: (1) repetition is necessary for knowledge acquisition, while paraphrasing helps only at smaller batch sizes; (2) with a fixed token budget, reallocating tokens from document repetition to auxiliary views improves learning, even counterintuitively for factual recall; (3) the effectiveness of auxiliary views does not depend on the strength of the teacher model that generates them; (4) the authors identify contextual and foundational forms of knowledge that aid learning when prior knowledge gaps exist; and (5) they examine how these effects manifest mechanistically through per-layer biases and compression. The authors conclude that auxiliary knowledge representations, which arise naturally in large pre-training corpora, are a key factor in pre-training success and offer an explanation for why data diversity matters.

论文概要

研究领域: NLP 作者: Joseph Lee, Yidi Huang, Dokyoon Kim 发布时间: 2026-09-06 arXiv: 2509.04288

中文摘要

关于大语言模型（LLM）在预训练期间如何获取知识，仍存在理解空白。我们假设辅助视角——即知识的重新表述——对学习具有因果帮助。我们设计了对照实验来分离这一效应。首先，我们确认了重复对知识获取是必要的，并明确改述仅在较小批次大小时有帮助。其次，在固定token预算下，将token从文档重复分配到辅助视角能提升学习效果，反直觉的是，这对事实回忆也成立。第三，辅助视角的有效性不依赖于生成它们的教师模型的强度。第四，我们识别了在有先验知识缺口时辅助学习的知识形式：情境性和基础性知识。最后，我们通过逐层偏差和压缩机制地检验了这些效应的表现方式。综合来看，我们的发现表明，辅助知识表示（在大规模预训练语料中自然产生）是预训练成功的关键因素，并为数据多样性为何重要提供了合理解释。

Original Abstract (English)

Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We posit that auxiliary views, reformulations of knowledge, are causally helpful for learning. We design controlled experiments to isolate this. First, we confirm that repetition is necessary for acquisition and clarify that paraphrasing helps only at smaller batch sizes. Second, holding the token budget fixed, allocating tokens from document repetition to auxiliary views improves learning, counterintuitively, even for factual recall. Third, the effectiveness of auxiliary views is not contingent on the strength of the teacher model that generates them. Fourth, we identify forms of knowledge, contextual and foundational, that aid learning in the presence of prior knowledge gaps. Final...

Key Findings

Repetition is necessary for knowledge acquisition; paraphrasing helps only at smaller batch sizes.
Fixed token budget: reallocating tokens from document repetition to auxiliary views improves learning—even for factual recall, which is counterintuitive.
Teacher model strength does not matter: the benefit of auxiliary views is independent of how strong the generating model is.
Contextual and foundational knowledge: these are the knowledge forms that help learning when prior knowledge gaps exist.
Mechanistic analysis: the effects are examined via per-layer biases and compression.

The authors conclude that auxiliary knowledge representations—which arise naturally in large pre-training corpora—are a key factor in pre-training success, and provide a rationale for why data diversity matters.

--- *自动采集于 2026-09-07*