> *"A teacher says a hundred sentences; a student doesn't need to memorize every one. The question is: which ones? CROP's answer — the ones whose answers change when a condition changes."*
The Problem with On-Policy Distillation
On-Policy Distillation (OPD) is a core post-training technique: the student generates responses, and the teacher scores each token, with the student learning from those scores. But the teacher scores every token — and not every token is worth learning. Some tokens are "key decision points" (they change when input conditions change); others are formulaic filler ("Solution:", "The answer is", punctuation) that stay the same no matter the input.
Existing selective OPD criteria (uncertainty, teacher-student disagreement) only ask: "does this token need to be learned better?" None ask: "is this token actually relevant to the task?" CROP (Counterfactual Relevance for On-Policy Distillation) fills that gap.
The Method: Counterfactual Triples
For each training sample, CROP constructs a triple:
- \(x_{para}\) (paraphrase): same meaning, different wording, e.g. "compute 3+5" → "what is the sum of three and five".
- \(x_{cf}\) (counterfactual): one task-relevant condition changed, e.g. "compute 3+5" → "compute 3×5".
- Strong baselines: CROP is compared against other *selective* OPD methods, not vanilla OPD — improving 2–3 points on an already-optimized setup is much harder than improving from scratch.
- Same training budget: gains come from picking better tokens, not more compute.
- Matched selection controls: CROP's top-ranked tokens beat random selection, which beats lowest-ranked tokens — evidence the relevance ranking genuinely captures valuable training positions.
- Removing paraphrase calibration (raw counterfactual sensitivity only): performance drops — calibration genuinely filters surface-form noise.
- Removing counterfactual sensitivity (paraphrase only): large drop — the core signal is "does the output change when conditions change?"
- Entropy interaction: CROP's gains are largest on mid-to-high entropy tokens and smallest on low-entropy ones — intuitive, since tokens the model is already certain about matter little either way. CROP further separates "uncertain because of task content" (worth learning) from "uncertain because of surface form" (not worth it).
- Pure OPD: no selection.
- Entropy: selects uncertain tokens — only "does it need learning?"
- TIP: a trained token-importance predictor.
- TA-OPD: uncertainty + teacher-student disagreement.
- CREDIT / CS-OPD: contrastive selective distillation.
- Triple construction quality: CROP depends on LLM-generated counterfactuals; noisy rewrites inject noise (the paper's Appendix A describes extensive validation/filtering).
- Compute overhead: training needs forward passes on all three inputs — roughly 3× vanilla OPD.
- Domain restriction: only math reasoning was tested; defining "task-relevant conditions" in open-domain dialogue is unclear.
- Magnitude: 1.92/2.96 points are modest in absolute terms; the gap vs unfiltered OPD may be smaller.
- Paper: arXiv:2608.13387
- Code: no public repository provided
- Training data: DAPO-Math-17K
The student generates a response \(y\) to \(x\). For each token position, CROP measures:
1. Counterfactual sensitivity: how much does the response distribution change when the input changes from \(x\) to \(x_{cf}\)? 2. Paraphrase sensitivity: how much does it change when the input changes from \(x\) to \(x_{para}\)?
CROP score = counterfactual sensitivity / paraphrase sensitivity.
The division is the key design: paraphrase sensitivity serves as a calibration baseline. If a position is sensitive even to meaning-preserving rewording, it captures surface form rather than task content. Dividing strips out surface-form sensitivity, leaving task-content sensitivity.
Results: +1.92 and +2.96 Points
CROP was tested in two teacher-student settings on DAPO-Math-17K (17,398 math problems; 16,594 kept after triple validation):
1. Qwen3-4B → Qwen3-1.7B: +1.92 points vs the strongest non-CROP selector 2. Qwen3-8B (GRPO) → Qwen3-4B: +2.96 points
Context for these numbers:
Ablations
Relation to Existing Methods
CROP's distinction: it is the first selector to treat task relevance as an independent dimension. The questions are orthogonal — a token may be "much needed" (uncertain) yet "task-irrelevant" (unchanged under counterfactuals). Traditional methods select it; CROP does not.
Why It Matters
1. Granularity alignment: optimization granularity should match the object being optimized. CROP applies this at token level, refining selection from sample-level to token-level and extending criteria from "optimization need" to "task relevance". 2. Engineering counterfactual reasoning: counterfactuals are a classic causal-inference tool but rarely used in LLM training due to cost. CROP builds triples offline, once, and reuses them — a tractable compromise. 3. Generalizable paraphrase calibration: calibrating sensitivity with meaning-preserving rewrites applies beyond distillation — to evaluation robustness, adversarial perturbation detection (task change vs. surface change), and data augmentation quality.