[论文] Measuring Task-Agnostic Training Data Influence Across Language Model ...

论文概要

研究领域: NLP
作者: Yuto Nishida, Hirokazu Kiyomaru, Yusuke Oda, Takashi Kodama, Chaoran Liu, Daisuke Kawahara, Yusuke Miyao, Max Müller-Eberstein, Masaru Isonuma
发布时间: 2026-08-13
arXiv: 2608.13515

中文摘要

在语言模型预训练中一致地测量训练数据影响具有挑战性。选择代表模型一般能力的下游任务或验证集很困难，且依赖中间检查点的任务性能使跨训练比较复杂化。我们提出一种训练数据影响度量，无需选择下游任务或验证集作为归因目标。具体而言，我们通过示例的梯度更新减少给定预训练运行最终参数平方距离的程度来定义其影响，并从中间检查点无需重新训练估计该量。将该方法应用于Pythia和PolyPythia套件的18个配置，我们发现影响数据的系统性时间变化。训练早期，与文学相关的数据更强烈地与朝向最终参数的轨迹对齐，而STEM数据在后期阶段变得更强对齐。这种定性交叉在模型配置间广泛一致。我们的结果提供了一种可行的轨迹级视角，观察影响数据如何在预训练过程中变化，补充了相对于特定下游任务或验证集定义的影响分析。

原文摘要

Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training. We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, we define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run, and estimate this quantity from intermediate checkpoints without retraining. Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data. Early in training, literature-related data are more strongly aligned with the trajectory toward the final parameters, whereas STEM data become more strongly aligned in later stages. This qualitative crossover is broadly consistent across model configurations. Our results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.

自动采集于 2026-08-15

#论文 #arXiv #NLP #小凯