Task-Agnostic Training Data Influence Across Language Model Pretraining

Forum topic · 小凯 · 2026-08-15

Summary

This paper addresses a core challenge in language model pretraining: measuring training data influence in a way that is consistent over the course of training, without depending on a particular downstream task or validation set. The authors define an example's influence as how much its gradient update reduces the squared distance to the final parameters of a pretraining run, and show how this quantity can be estimated from intermediate checkpoints without retraining. Applied to 18 configurations from the Pythia and PolyPythia suites, the analysis reveals systematic temporal shifts in which data dominate the trajectory. Early in training, literature-related data align more strongly with movement toward the final parameters, while STEM data become more strongly aligned in later stages. This qualitative crossover holds broadly across model configurations. The work provides a tractable, trajectory-level view of how influential data evolve during pretraining, complementing existing influence analyses that are tied to specific downstream tasks or validation sets.

Paper Overview

Research Area: NLP

Authors: Yuto Nishida, Hirokazu Kiyomaru, Yusuke Oda, Takashi Kodama, Chaoran Liu, Daisuke Kawahara, Yusuke Miyao, Max Müller-Eberstein, Masaru Isonuma

Release Date: 2026-08-13

arXiv: 2608.13515

Abstract

Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training. We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, we define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run, and estimate this quantity from intermediate checkpoints without retraining. Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data. Early in training, literature-related data are more strongly aligned with the trajectory toward the final parameters, whereas STEM data become more strongly aligned in later stages. This qualitative crossover is broadly consistent across model configurations. Our results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.

Chinese Summary (中文摘要)

在语言模型预训练中一致地测量训练数据影响具有挑战性。选择代表模型一般能力的下游任务或验证集很困难，且依赖中间检查点的任务性能使跨训练比较复杂化。我们提出一种训练数据影响度量，无需选择下游任务或验证集作为归因目标。具体而言，我们通过示例的梯度更新减少给定预训练运行最终参数平方距离的程度来定义其影响，并从中间检查点无需重新训练估计该量。将该方法应用于Pythia和PolyPythia套件的18个配置，我们发现影响数据的系统性时间变化。训练早期，与文学相关的数据更强烈地与朝向最终参数的轨迹对齐，而STEM数据在后期阶段变得更强对齐。这种定性交叉在模型配置间广泛一致。我们的结果提供了一种可行的轨迹级视角，观察影响数据如何在预训练过程中变化，补充了相对于特定下游任务或验证集定义的影响分析。

Key Points

Problem: Existing influence analysis in NLP pretraining depends on a chosen downstream task or validation set, which may not represent general capabilities and complicates checkpoint-to-checkpoint comparison.
Method: Define a training example's influence by how much its gradient update reduces the squared distance to the final parameters of a pretraining run; estimate this contribution from intermediate checkpoints without retraining.
Scope: Evaluated on 18 configurations from the Pythia and PolyPythia suites.
Finding: Influential data evolve systematically over training. Literature-related data dominate the trajectory early, while STEM data dominate later.
Consistency: The literature-to-STEM crossover is broadly consistent across model configurations.
Contribution: A trajectory-level, task-agnostic view of data influence that complements task- or validation-set-based attribution methods.

--- *Auto-collected on 2026-08-15*