Overview
- Research areas: cs.AI, cs.IR, cs.LG
- Authors: Shiqiang Wang, Herbert Woisetschläger, Hans Arno Jacobsen
- arXiv: 2505.01250
- Paper: https://arxiv.org/abs/2505.01250
Abstract (translated)
Data is fundamental to large language models (LLMs). However, understanding what makes certain data useful for different stages of an LLM workflow—including training, tuning, alignment, and in-context learning—and why, remains an open question. Current approaches rely heavily on extensive experimentation with large public datasets to obtain empirical heuristics for data filtering and dataset construction. These approaches are compute intensive and lack a principled way of understanding the essence of how specific data characteristics drive LLM behavior.
In this position paper, the authors advocate for developing systematic methodologies for generating synthetic sequences from appropriately defined random processes, with the goal that these sequences can reveal useful characteristics when used in one or multiple stages of the LLM workflow. They refer to such sequences as data probes.
By observing LLM behavior on data probes, researchers can systematically study how data characteristics influence model performance, generalization, and robustness. The probing sequences exhibit statistical properties that can be viewed through theoretical concepts, such as typical sets, generalized to describe the behaviors of LLMs. This data-probe approach provides a pathway for uncovering foundational insights into the role of data in LLM training and inference, beyond empirical heuristics.