Overview
Research area: NLP Authors: Hang Zhang, Warren J. Gross Published: 2026-07-20 arXiv: 2607.18199 Categories: cs.CL, cs.LG
Summary
Not all training samples contribute equally to large language model fine-tuning. Selecting informative training samples can reduce computational cost while preserving downstream performance. Many existing data selection methods rely on indirect heuristics such as data quality, diversity, or reasoning trajectory length. However, the effectiveness of these fixed criteria is task-dependent and hard to generalize across diverse downstream tasks.
Perplexity-based data selection offers a simple, model-aware way to estimate sample difficulty, but existing approaches typically score the entire training sequence, ignoring the differences between the learning objectives of language modeling and reasoning tasks.
This paper proposes PPL-Factory, a simple and interpretable data selection framework that combines task-aware perplexity scoring with budget-aware selection criteria.
Key Results
- On GSM8K, PPL-Factory outperforms other state-of-the-art data selection methods using only 1% of the training set.
- With 10% of the data, PPL-Factory exceeds full-data fine-tuning accuracy by 0.9 points on GSM8K and 4.8 points on MATH.
Original Abstract (Condensed)
> We propose PPL-Factory, combining task-aware perplexity-based scores and budget-aware selection. Using only 1% of training data, it outperforms SOTA methods on GSM8K. With 10% data, it exceeds full-data fine-tuning on GSM8K and MATH.
--- *Auto-collected on 2026-07-22*