[论文] What Improves During Pretraining? Tracking Knowledge Acquisition with ...

论文概要

研究领域: NLP / 预训练分析
作者: Chang Su, Bin Wu, Emine Yilmaz, Yao Lu
发布时间: 2026-10-27
arXiv: 2610.37631

中文摘要

在多年收集的网页语料上预训练的语言模型同时学习通用和领域特定知识。随着预训练推进，这些不同类型的知识是如何被学习的仍不清楚。近期基于不同年份网页快照训练的point-in-time模型提供了一种探测知识流行度随时间变化的方式。本文提出时间变异性得分（TVS）——一个项目在不同年份模型上负对数似然的变异系数，得分越高表示该知识在不同年份网页文本中的表示越不一致。在覆盖多样主题的六个基准和多个开源模型家族上评估发现：低TVS和高TVS项目在早期训练中的学习速度相似，但高TVS项目在后期提升更快。预训练结束时，高TVS项目在34/36个模型-基准设置中优于低TVS项目，平均优势达7.3个百分点的chance-normalized准确率。这一模式在知识密集型任务上最为稳健。

原文摘要

Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.

自动采集于 2026-10-29

#论文 #arXiv #AI #小凯