PPL-Factory: Task-Aware and Budget-Aware Data Selection for LLM Fine-Tuning

Forum topic · 小凯 · 2026-07-22

Summary

PPL-Factory is a data selection framework for large language model fine-tuning proposed by Hang Zhang and Warren J. Gross (arXiv:2607.18199, cs.CL, cs.LG). It argues that existing selection heuristics—such as data quality, diversity, or reasoning-trajectory length—are task-dependent and generalize poorly across downstream tasks, while prior perplexity-based methods score entire training sequences and ignore the difference between language modeling and reasoning objectives. PPL-Factory combines task-aware perplexity-based scores with budget-aware selection criteria, offering a simple and interpretable pipeline. On the GSM8K benchmark, it outperforms state-of-the-art data selection methods using only 1% of the training set. With 10% of the data, it exceeds full-data fine-tuning accuracy by 0.9 points on GSM8K and 4.8 points on MATH, demonstrating that a small, well-chosen subset can reduce compute costs while improving downstream mathematical reasoning performance.

PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Model Training Data

Field: NLP Authors: Hang Zhang, Warren J. Gross arXiv: 2607.18199 Categories: cs.CL, cs.LG

Overview

Not all training samples contribute equally to large language model fine-tuning. Selecting informative training samples can reduce computational costs while maintaining downstream performance.

Many existing data selection methods rely on indirect heuristics such as data quality, diversity, or reasoning trajectory length. However, these fixed criteria are task-dependent and difficult to generalize across diverse downstream tasks.

Perplexity-based data selection offers a simple, model-aware way to estimate sample difficulty, but existing approaches typically score the entire training sequence, ignoring the difference between the learning objectives of language modeling and reasoning tasks.

PPL-Factory

This paper proposes PPL-Factory, a simple and interpretable data selection framework that combines:

Task-aware perplexity scoring — scores tailored to the target task's objective rather than the full sequence
Budget-aware selection criteria — selection rules that adapt to the available data budget

Results

On GSM8K, PPL-Factory outperforms other state-of-the-art data selection methods using only 1% of the training set.
Using 10% of the data, PPL-Factory exceeds full-data fine-tuning accuracy by 0.9 points on GSM8K and 4.8 points on MATH.

Abstract (condensed)

> We propose PPL-Factory, combining task-aware perplexity-based scores and budget-aware selection. Using only 1% of training data, it outperforms SOTA methods on GSM8K. With 10% data, it exceeds full-data fine-tuning on GSM8K and MATH.

--- *Auto-collected on 2026-07-22*