The Pareto Truth of RL Training Data: Why 84% of Samples Can Be Discarded

Forum topic · 小凯 · 2026-05-11

Summary

This analysis of the LIMR paper (Li et al., 2025, arXiv:2502.11886) shows that most reinforcement learning (RL) training data contributes little to learning. By tracking each sample's reward trajectory during PPO training on Qwen2.5-Math-7B, the authors classify samples into 'rock' (never solved, ~20-30%), 'ice' (solved immediately, ~30-40%), and 'fuel' (growing reward, ~30-40%) types. Only fuel-type samples drive capability growth. Their proposed Learning Impact Measurement (LIM) score, which measures alignment between a sample's reward trajectory and the average trajectory, filters MATH-FULL (8,523 problems) down to just 1,389 samples (16.3%). This curated subset matches or exceeds full-data performance: 78.0% vs 76.6% on MATH500, 63.8% vs 61.9% on AMC23, and 32.5% on AIME24, while random sampling of the same size loses ~8 points. LIMR also more than doubles AIME24 accuracy (32.5% vs 15.8%) compared to same-scale SFT distillation methods LIMO and s1, suggesting RL on capacity-matched data beats imitation of oversized teacher outputs. The post argues the industry should shift from a scale race to a density race, while noting limitations: validation is confined to math reasoning and 7B models, and static filtering may miss samples that become useful later in training.

This post re-analyzes the experimental material of LIMR: Less Is More for RL Scaling (Li, Zou & Liu, SJTU/SII/GAIR, arXiv:2502.11886) through the lens of information density — the number of actionable insights per unit of token/text. Code and data: https://github.com/GAIR-NLP/LIMR

Key points

Core claim: In RL training, data volume is not the bottleneck — value density is. 1,389 carefully selected samples beat the full 8,523-sample dataset.
Method: Learning Impact Measurement (LIM) scores each sample by how well its reward trajectory aligns with the average trajectory across training.
Result: With threshold θ = 0.6, LIM keeps only 16.3% of MATH-FULL yet matches or exceeds full-data accuracy (MATH500: 78.0% vs 76.6%; AMC23: 63.8% vs 61.9%; AIME24: 32.5% equal).
Paradigm contrast: LIMR (RL + curated data) more than doubles AIME24 accuracy vs same-scale SFT distillation methods LIMO and s1 (32.5% vs 15.8%).
So-what: The industry should shift from a "scale race" to a "density race."

1. Data layer: sample value distribution in MATH-FULL

Not all training samples participate in learning. By tracking per-sample reward trajectories on MATH-FULL (8,523 problems, difficulty Level 3–5), three archetypes emerge:

| Type | Reward trajectory | Estimated share | Teaching value | |:---:|:---|:---:|:---:| | 🪨 Rock (constant zero) | $r \approx 0$ throughout; model never solves it | ~20–30% | Zero — beyond capability boundary | | 🧊 Ice (saturated) | $$r = 1$$ early, then unchanged | ~30–40% | Zero — already mastered | | 🔥 Fuel (growing) | Reward rises with training | ~30–40% | High — extends capability boundary |

Rock and ice samples are "dead weight": they consume GPU memory and compute without contributing gradient signal.

The LIM score automates identification of fuel-type samples:

s_i = 1 - \frac{\sum_{k=1}^{K}(r_i^k - r_{\text{avg}}^k)^2}{\sum_{k=1}^{K}(1 - r_{\text{avg}}^k)^2}

Geometrically, this measures the (normalized) Euclidean distance between a sample's trajectory vector and the average trajectory vector, mapped to $(-\infty, 1]$ :

| LIM Score | Meaning | Sample type | |:---:|:---|:---| | $s_i \approx 1$ | Trajectory nearly coincides with the average | Fuel | | $s_i \approx 0$ | Moderate deviation | Borderline | | $s_i \ll 0$ | Opposite or unrelated trajectory | Rock / ice |

With $\theta = 0.6$ , 8,523 problems shrink to 1,389 (16.3%) — systematic removal of dead weight, not random subsampling.

2. Method layer: selection strategies compared

| Strategy | Logic | Samples | AIME24 | MATH500 | AMC23 | |:---:|:---|:---:|:---:|:---:|:---:| | FULL | No filtering | 8,523 | 32.5% | 76.6% | 61.9% | | RAND | Random sampling | 1,389 | 25.8% | 66.0% | 56.3% | | LINEAR | Monotonically improving samples | 1,138 | 28.3% | 74.6% | 61.9% | | LIMR | Trajectory-alignment scoring | 1,389 | 32.5% | 78.0% | 63.8% |

Three takeaways:

1. RAND is far worse than FULL (−8.1% average): being "less" is not an advantage — *precisely less* is. 2. LINEAR is close to FULL (−2%): samples with learning signals do matter, but monotonic-increase heuristics miss good samples that improve fast then plateau. 3. LIMR beats FULL: low-value samples in the full set appear to cause negative interference — slowing convergence or injecting noisy gradients.

Training dynamics (paper Figure 3) corroborate this: LIMR's accuracy rises faster then matches FULL; its sequence-length curve (steadily growing after an early dip) and reward curve (approaching 1.0 faster) are smoother than FULL's, and clearly better than RAND's.

3. Comparison layer: RL curation vs SFT distillation

| Method | Paradigm | Samples | Data source | AIME24 | MATH500 | AMC23 | |:---:|:---:|:---:|:---|:---:|:---:|:---:| | LIMR | RL + curated | 1,389 | Filtered from MATH-FULL | 32.5% | 78.0% | 63.8% | | LIMO | SFT + distilled | 817 | Generated by a 32B model | 15.8% | 65.0% | 56.3% | | s1 | SFT + distilled | 1,000 | Generated by a 32B model | 15.8% | 55.8% | 42.5% |

The gap on AIME24 is over 2×. The proposed mechanism:

SFT is imitation learning: a 7B student copying 32B teacher chains-of-thought. If the required reasoning exceeds the student's representational capacity, imitation degrades into rote memorization.
RL is boundary exploration: the model keeps only strategies that succeed. Fuel-type samples sit in the model's *zone of proximal development* — hard enough to drive improvement, easy enough for positive reward to accumulate.

Distillation is not a panacea: when the teacher–student capability gap is too large, "knowledge transfer" degenerates into "ineffective copying."

4. System layer: data composition → training dynamics

Full-data training suffers from high within-batch variance: gradient updates mix strong-signal fuel samples with zero-signal dead weight. This causes:

1. Noisy gradient directions — zero-reward samples produce uninformative gradients that dilute the optimization direction. 2. Difficult value estimation — the critic struggles with high-variance batches. 3. Length-policy oscillation — random length distributions from dead-weight samples disrupt learning of reasonable reasoning lengths.

Filtering lowers batch variance:

\text{Var}(\nabla_\theta J) \propto \text{Var}(r_i) \cdot \text{sample count}

With reward distributions concentrated and matched to model capability, policy-gradient variance drops and optimization smooths — while retained diversity avoids premature convergence.

5. Limitations and structural recommendations

Open questions:

1. Cross-domain generalization: LIMR is validated only on math. Do fuel-type distributions exist in code generation, scientific QA, or multimodal reasoning, where reward functions differ greatly? 2. Model-scale effects: at 32B–70B, representational capacity covers more problems — does the dead-weight share shrink, narrowing the gap between full and curated data? 3. LIM's static nature: samples scored as "rock" early in training might become "fuel" later as the model enters new capability stages. Does static filtering miss delayed teaching value?

From a scale race to a density race:

| Dimension | Old paradigm | New paradigm | |:---:|:---|:---| | Core metric | Dataset size | Information density (LIM score distribution) | | Engineering focus | Collection & cleaning pipelines | Value assessment & dynamic filtering systems | | Cost structure | Annotation + storage + compute | Trajectory tracking + scoring + curated training | | Competitive edge | "We used 150K samples" | "Our 1.4K beat their 8.5K" | | Reproducibility | Low (scale hard to match) | High (standardizable filtering algorithms) |

Paper details (verified)

| Item | Content | |:---:|:---| | Title | LIMR: Less Is More for RL Scaling | | Authors | Xuefeng Li, Haoyang Zou, Pengfei Liu (SJTU / SII / GAIR) | | arXiv | 2502.11886, published 2025-02-17 | | Code/data/model | https://github.com/GAIR-NLP/LIMR | | Model / algorithm | Qwen2.5-Math-7B; PPO (OpenRLHF; rollout batch 1,024, 8 samples/prompt, temp=1.2, actor lr=5e-7, critic lr=9e-6, KL=0.01) | | Data | MATH-FULL (8,523 problems, Level 3–5) → LIMR (1,389 problems, θ=0.6) | | Benchmarks | MATH500; AIME2024 (30 problems, 4 samples, temp=0.4); AMC2023 (40 problems, 4 samples) | | Key results | MATH500 78.0% > FULL 76.6%; AMC23 63.8% > 61.9%; AIME24 tied at 32.5%. AIME24 >100% higher than SFT methods LIMO/s1. Baselines: RAND (−8.1%), LINEAR (−2%) |