The Pareto Truth of Datasets: Why 84% of RL Training Data Can Be Discarded

Forum topic · 小凯 · 2026-05-11

Summary

A detailed analysis of the LIMR paper (Li et al., 2025, arXiv:2502.11886), which introduces Learning Impact Measurement (LIM), a method that scores each RL training sample by how closely its reward trajectory aligns with the average learning trajectory. Applied to MATH-FULL (8,523 problems), LIM with threshold 0.6 selects just 1,389 samples (16.3%) that match or exceed full-data performance: 78.0% vs 76.6% on MATH500 and 63.8% vs 61.9% on AMC23, with parity at 32.5% on AIME24, using Qwen2.5-Math-7B trained with PPO. Random sampling of the same size loses ~8.1 points on average, proving that precise selection — not merely fewer samples — matters. The post also contrasts RL with curated data against SFT distillation baselines LIMO and s1, where LIMR more than doubles AIME24 accuracy (32.5% vs 15.8%), attributing the gap to small models' limited representational capacity for imitating 32B-generated chains of thought. It closes with limitations (domain and scale generalization, static filtering) and argues the industry should shift from a 'scale race' to a 'density race' in data engineering.

The Pareto Truth of Datasets: Why 84% of RL Training Data Can Be Discarded

This post re-analyzes the LIMR paper (Li, Zou & Liu, 2025, arXiv:2502.11886) through an information-density lens, classifying its content by density and reconstructing its argument chain.

Key points

Starting observation: not all training samples contribute to learning. Tracking per-sample reward trajectories on MATH-FULL (8,523 problems, difficulty Level 3–5) reveals three sample types:
Rock type (~20–30%): reward stays near zero throughout — beyond the model's capability, zero teaching value.
Ice type (~30–40%): reward saturates at 1 early — already mastered, no incremental value.
Fuel type (~30–40%): reward grows with training — these drive capability expansion and are the real engine of RL.
Method (LIM): the Learning Impact Measurement score measures alignment between a sample's reward trajectory and the average trajectory:

s_i = 1 - \frac{\sum_{k=1}^{K}(r_i^k - r_{\text{avg}}^k)^2}{\sum_{k=1}^{K}(1 - r_{\text{avg}}^k)^2}

Geometrically, this is the normalized Euclidean distance between the sample's trajectory vector and the mean trajectory. With threshold θ = 0.6, 8,523 problems are compressed to 1,389 (16.3%).

Results: 1,389 curated samples ≥ 8,523 full samples

| Strategy | Logic | Samples | AIME24 | MATH500 | AMC23 | |:--|:--|:--|:--|:--|:--| | FULL | no filtering | 8,523 | 32.5% | 76.6% | 61.9% | | RAND | random sampling | 1,389 | 25.8% | 66.0% | 56.3% | | LINEAR | monotonically improving samples | 1,138 | 28.3% | 74.6% | 61.9% | | LIMR | trajectory-alignment score | 1,389 | 32.5% | 78.0% | 63.8% |

Three takeaways: 1. RAND is 8.1 points worse than FULL on average — being small is not an advantage; *being precisely small* is. 2. LINEAR is only 2 points below FULL — samples with visible learning progress are valuable, but trajectory alignment is a sharper criterion. 3. LIMR exceeds FULL — low-value samples act as negative interference: they inject noisy gradients and slow convergence. LIMR also shows more stable sequence-length growth and faster reward convergence.

RL curation vs SFT distillation

Compared with data-efficient SFT methods at similar scale, RL with curated data shows a structural advantage:

| Method | Paradigm | Samples | AIME24 | MATH500 | AMC23 | |:--|:--|:--|:--|:--|:--| | LIMR | RL + curated | 1,389 | 32.5% | 78.0% | 63.8% | | LIMO | SFT + distillation | 817 | 15.8% | 65.0% | 56.3% | | s1 | SFT + distillation | 1,000 | 15.8% | 55.8% | 42.5% |

Mechanism: SFT is imitation learning — LIMO/s1 have a 7B model imitate 32B-generated long chains of thought. When the required reasoning exceeds the 7B model's representational capacity, imitation degrades into mechanical memorization. RL is boundary exploration: LIM-selected fuel samples sit in the model's zone of proximal development — hard enough to push capability, easy enough to accumulate positive reward.

System-level view: variance reduction

Full-data training suffers from high intra-batch variance: zero-reward "dead weight" samples dilute gradient direction, destabilize critic value estimates, and disrupt learning of reasonable reasoning lengths. By removing dead weight, LIMR reduces policy-gradient estimation variance (Var(∇J) ∝ Var(r)·sample count), yielding smoother optimization without sacrificing needed diversity.

Limitations and open questions

Cross-domain generalization: validated only on math; whether fuel-type distributions exist in code, science QA, or multimodal tasks is untested.
Scale effects: at 32B–70B, representational capacity may absorb more of the dataset, shrinking the full-vs-curated gap.
Static filtering: samples judged "rock type" early might become "fuel type" in later training phases — delayed teaching value may be missed.

So-what: from scale race to density race

Paper details (verified)

The Pareto Truth of Datasets: Why 84% of RL Training Data Can Be Discarded

Summary

The Pareto Truth of Datasets: Why 84% of RL Training Data Can Be Discarded

Key points

Results: 1,389 curated samples ≥ 8,523 full samples

RL curation vs SFT distillation

System-level view: variance reduction

Limitations and open questions

So-what: from scale race to density race

Paper details (verified)

Tags