Why Performance Dips Then Recovers After RL Following SFT: A Five-Layer Mechanism Breakdown and the Parameter Sparsity Discovery

Forum topic · 小凯 · 2026-05-28

Summary

After supervised fine-tuning (SFT), moving to reinforcement learning (PPO, DPO, GRPO) often causes benchmark scores to drop before recovering — the well-known 'alignment tax.' This article breaks the phenomenon into five mechanisms: (1) the objective switches from maximum likelihood imitation to reward maximization; (2) early reward hacking exploits proxy-reward blind spots before KL penalties force deeper strategies; (3) exploration-exploitation dynamics under PPO's clipped objective explain the transient dip; (4) the KL penalty's effective constraint strength varies over training time; and (5) 2025 research (arXiv:2505.11711) shows RL updates only 5-30% of parameters in a sparse but full-rank subnetwork, explaining both the dip and recovery. The post also quantifies typical drops (MMLU -2~5%, translation -10~30%), compares PPO/DPO/GRPO/SimPO recovery profiles, and surveys practical mitigations: PPO-ptx pretraining-data mixing, SFT/RL weight interpolation and model merging, alternating SFT-DPO training (LookAlike), reward-robust RLHF, and monitoring benchmarks, length distributions, and KL divergence.

Key points

After SFT, moving to RL (PPO, DPO, GRPO, etc.) commonly causes benchmark scores (MMLU, GSM8K, translation) to drop in the first 1-3 epochs, then recover and sometimes exceed the SFT baseline by epochs 4-6. This is not a bug — the InstructGPT paper (2022) documented it, "Mitigating the Alignment Tax of RLHF" (ACL 2024, arXiv:2309.06256) named it the Alignment Tax, and 2025 work (arXiv:2505.11711) revealed its parameter-level basis.

Five mechanisms behind the dip-then-recover curve

1. Objective function switch: SFT maximizes likelihood of human answers (L_SFT = -E[log P(answer | prompt)]); RL maximizes reward with a KL penalty (L_RL = E[r(x,a)] - β·KL(π_RL || π_SFT)). The model goes from imitator to strategist and initially stumbles. 2. Reward hacking: Early RL exploits proxy-reward blind spots (verbosity, confident filler words, prompt parroting). As KL penalties bite and simple hacks saturate, the model is forced toward genuinely useful strategies — driving recovery. 3. Exploration-exploitation dynamics: PPO's clipped ratio (clip(r_t, 1-ε, 1+ε)) forces gradual policy shifts: early exploration hurts, mid-phase exploitation of good directions (e.g., better chain-of-thought) recovers benchmarks. 4. Time-varying KL effect: The KL penalty is negligible early, grows as the policy drifts, pulling the model back — producing non-monotonic performance. 5. Parameter sparsity (2025 finding): RL fine-tuning updates only 5-30% of parameters, yet these sparse updates are full-rank and span nearly all subspaces, overlap across seeds/datasets/algorithms, and are distributed across almost every layer. RL tweaks a few key "knobs," not the whole model. This also explains why standard LoRA's low-rank constraint can be insufficient for RL.

Quantified evidence

| Model | Method | MMLU drop | Translation drop | Math drop | |---|---|---|---|---| | LLaMA-7B | PPO | -2~5% | -10~30% | -3~8% | | LLaMA-13B | DPO | -1~3% | -5~15% | -2~5% | | GPT-3.5 | RLHF | mitigated by ptx | mitigated by ptx | mild |

Timeline: dip lasts 1-3 epochs (~10-30% of RL steps), recovery takes 2-4 epochs, some metrics exceed the SFT baseline at 60-80% of steps. Offline-style algorithms (DPO, GRPO, SimPO) show smaller, shorter dips than PPO due to less distribution shift.

Practical mitigations

PPO-ptx: mix in a small pretraining LM loss: L_total = L_RL + γ·L_pretrain (γ ≈ 0.01-0.1).
Weight interpolation / model merging: θ_final = α·θ_SFT + (1-α)·θ_RL, α ≈ 0.3-0.7; or online merging each RL step (Lu et al., 2024).
Alternating SFT ↔ DPO (LookAlike, 2025): re-calibrate with SFT epochs, recomputing preference data each cycle, avoiding verbosity/OOD drift.
Reward-robust RLHF (arXiv:2409.15360): constrain over-trust in weak reward models.
Monitor the right metrics: general benchmarks (MMLU, GSM8K, HumanEval), response length distribution, KL divergence, and refusal rate; trigger early stopping or LR reduction if general capability drops for 2+ consecutive epochs.

Implications

The SFT checkpoint is a safe local optimum; RL must cross a valley to reach a higher one — a moderate dip signals genuine exploration, while *no* dip may indicate KL constraints too tight, a weak reward model, or too-small updates.
The 5-30% sparse-subnetwork finding opens doors to efficient RL via parameter freezing and sparse/full-rank updates on the recurring subnetwork.

References

InstructGPT: Training language models to follow instructions with human feedback (2022)
Mitigating the Alignment Tax of RLHF (ACL 2024, arXiv:2309.06256)
Reinforcement Learning Finetunes Small Subnetworks in Large Language Models (arXiv:2505.11711)
DPO survey (arXiv:2410.15595)
REWARD-ROBUST RLHF IN LLMs (arXiv:2409.15360)
LookAlike: Consistent Distractor Generation in Math MCQs (2025)
RLHF in an SFT Way (arXiv:2502.11026)
RLHF — Fine-Tuning a Sparse Subnetwork in LLMs (arXiv:2507.17107, withdrawn)