Paper Overview
Field: NLP Authors: Yuto Nishida, Hirokazu Kiyomaru, Yusuke Oda, Takashi Kodama, Chaoran Liu, Daisuke Kawahara, Yusuke Miyao, Max Müller-Eberstein, Masaru Isonuma arXiv: 2608.13515Summary
Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training.The authors propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, an example's influence is defined by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run. This quantity is estimated from intermediate checkpoints without retraining.
Key Findings
- Applied to 18 configurations from the Pythia and PolyPythia suites
- Systematic temporal changes in influential data were observed:
- Early in training: literature-related data align more strongly with the trajectory toward the final parameters
- Later stages: STEM data become more strongly aligned
- This qualitative crossover is broadly consistent across model configurations
Significance
The results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.--- *Auto-collected on 2026-08-15*