Position: Let's Develop Data Probes to Fundamentally Understand How Data Affects LLM Performance

论文概要

研究领域: cs.AI, cs.IR, cs.LG 作者: Shiqiang Wang, Herbert Woisetschläger, Hans Arno Jacobsen 发布时间: 2026-05-21 arXiv: 2505.01250

中文摘要

数据是大语言模型（LLM）的根本。然而，理解什么样的数据对LLM工作流的不同阶段（包括训练、微调、对齐、上下文学习等）有用，以及为什么有用，仍然是一个开放问题。当前方法主要依赖对大型公共数据集进行大量实验来获得数据过滤和数据集构建的经验启发式方法。这些方法计算密集，且缺乏一种原则性的方式来理解特定数据特征如何驱动LLM行为的本质。在本立场论文中，我们倡导开发系统化的方法论，从适当定义的随机过程中生成合成序列，目标是这些序列在用于LLM工作流的一个或多个阶段时能揭示有用特征。我们将此类序列称为「数据探针」。通过观察LLM在数据探针上的行为，研究人员可以系统地研究数据特征如何影响模型性能、泛化能力和鲁棒性。探针序列展现的统计性质可以使用理论概念（如典型集）来观察，这些概念被推广用于描述LLM的行为。这种数据探针方法为揭示数据在LLM训练和推理中作用的基础性见解提供了一条路径，超越了经验启发式方法。

原文摘要

Data is fundamental to large language models (LLMs). However, understanding of what makes certain data useful for different stages of an LLM workflow, including training, tuning, alignment, in-context learning, etc., and why, remains an open question. Current approaches rely heavily on extensive experimentation with large public datasets to obtain empirical heuristics for data filtering and dataset construction. These approaches are compute intensive and lack a principled way of understanding the essence of how specific data characteristics drive LLM behavior. In this position paper, we advocate for the need of developing systematic methodologies for generating synthetic sequences from appropriately defined random processes, with the goal that these sequences can reveal useful characteristics when they are used in one or multiple stages of the LLM workflow. We refer to such sequences as data probes. By observing LLM behavior on data probes, researchers can systematically conduct studies on how data characteristics influence model performance, generalization, and robustness. The probing sequences exhibit statistical properties that can be viewed using theoretical concepts, such as typical sets, which are generalized to describe the behaviors of LLMs. This data-probe approach provides a pathway for uncovering foundational insights into the role of data in LLM training and inference, beyond empirical heuristics.

自动采集于 2026-05-21

#论文 #arXiv #AI #小凯