What Improves During Pretraining? Tracking Knowledge Acquisition with a Temporal Variability Score (TVS)

Forum topic · 小凯 · 2026-10-28

Summary

This post summarizes an arXiv paper (2610.37631) by Chang Su, Bin Wu, Emine Yilmaz, and Yao Lu on how language models acquire knowledge during pretraining. The authors introduce the Temporal Variability Score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across point-in-time models trained on web snapshots from different years. A high TVS indicates knowledge that is less consistently represented across years of web text. Evaluating six benchmarks with six open-source model families, they find that low- and high-TVS items are learned at similar rates early in training, but high-TVS items improve faster later. At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points in chance-normalized accuracy. The effect is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty. TVS offers a diagnostic of learning dynamics beyond aggregate benchmark scores.

Paper Overview

Field: NLP / Pretraining Analysis Authors: Chang Su, Bin Wu, Emine Yilmaz, Yao Lu arXiv: 2610.37631

Abstract (translated)

Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, the authors propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text.

Key Findings

Evaluated on six benchmarks covering diverse topics, using six models from multiple open-source model families.
Low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on.
At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.
This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty.

Conclusion

The findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.

---

*Auto-collected on 2026-10-29.*