SFT-to-RL Performance Dips Before Recovering: Five Mechanisms Explained, Plus the Parameter Sparsity Finding

Forum topic · 小凯 · 2026-05-28

Summary

When large language models transition from supervised fine-tuning (SFT) to reinforcement learning (PPO, DPO, GRPO), benchmark scores typically drop in early RL epochs—translation and math abilities can fall sharply—before recovering and sometimes exceeding the SFT baseline. This article analyzes this 'alignment tax' phenomenon across five mechanisms: (1) the objective-function switch from likelihood-based imitation to reward maximization; (2) early-stage reward hacking followed by deeper strategies as KL penalties and reward-model saturation take effect; (3) the exploration-exploitation dynamic constrained by PPO's clipped objective; (4) the time-varying effective strength of the KL divergence penalty; and (5) 2025 research (arXiv:2505.11711) showing RL updates only 5–30% of parameters, yet in a full-rank manner spanning nearly all subspaces. The post also quantifies alignment tax magnitudes, compares algorithms (PPO shows the largest, slowest-recovering dips; GRPO/SimPO the smallest), and offers practical mitigation strategies: PPO-ptx pretraining-mix loss, weight interpolation between SFT and RL checkpoints, alternating SFT-DPO training, reward robustness constraints, and proper monitoring metrics.

This post explains why benchmark performance typically drops first, then recovers when transitioning from SFT to RL fine-tuning (PPO, DPO, GRPO), and what the 2025 parameter-sparsity finding means for training practice.

The phenomenon

During SFT, loss falls steadily and benchmarks climb. Upon entering RL, the first epochs often show MMLU dropping ~3 points, GSM8K ~5 points, and translation capability halving. After 3–6 epochs, scores recover and sometimes exceed the SFT baseline.
This was documented in OpenAI's InstructGPT paper (2022): *"Just using RL objective leads to performance degradation on many NLP tasks."* Their fix was PPO-ptx—an auxiliary pretraining-data LM loss added to the RL objective.
"Mitigating the Alignment Tax of RLHF" (ACL 2024, arXiv:2309.06256) systematically tracked the effect: reward scores rise while general capabilities fall; common-sense reasoning rises then falls; catastrophic forgetting is the essence of the alignment tax.
A 2025 UIUC paper ("Reinforcement Learning Finetunes Small Subnetworks in Large Language Models", arXiv:2505.11711) observed directly: *"RL fine-tuning effectively modifies only a small subnetwork (typically 5–30% of parameters), leaving the majority of weights essentially unchanged."*

Five-layer mechanism breakdown

1. Objective function switch: SFT optimizes L_SFT = -E[log P(answer | prompt)] (maximum likelihood, imitation). RL optimizes L_RL = E[r(x,a)] - β * KL(π_RL || π_SFT) (reward maximization). The model shifts from imitator to strategist—and early exploration is costly. 2. Reward hacking: Early RL exploits reward-model blind spots (verbosity, over-confident words, prompt echoing). As KL penalties bind and the reward model saturates, these shortcuts stop working, forcing genuinely better reasoning chains—this drives the recovery. 3. Exploration–exploitation dynamics: PPO's clipped surrogate objective (clip(r_t, 1-ε, 1+ε)) restricts policy jumps. Early = unconstrained-feeling exploration (dips), middle = exploitation of good directions (recovery), late = Pareto balance between reward and KL constraint. 4. Time-varying KL effect: The coefficient β is fixed, but actual KL grows over training—early the penalty is ~0 (free exploration), mid-training it pulls the policy back, late it stabilizes. This produces non-monotonic performance. 5. Parameter update sparsity: Only 5–30% of parameters change, but the updates span almost all subspaces and are full-rank; different seeds, datasets, and algorithms (PPO/DPO/GRPO) update significantly overlapping subnetworks. SFT leaves a highly optimized system; RL just re-tunes a few key knobs. This also explains why LoRA often underperforms in RL—its low-rank constraint may cut off the needed full-rank update paths.

Quantitative evidence

Alignment tax magnitude (approximate): LLaMA-7B + PPO: MMLU −2~5%, translation −10~30%, math −3~8%; LLaMA-13B + DPO: smaller dips; GPT-3.5 RLHF: mitigated by ptx.
Typical timeline: decline lasts 1–3 RL epochs, recovery 2–4 epochs, with some metrics surpassing the SFT baseline at 60–80% of total steps.
Algorithm comparison: PPO shows the largest/slowest dips; DPO moderate; GRPO and SimPO smallest and fastest-recovering (offline algorithms have less distribution shift).

Engineering mitigations

1. Pretraining mix (PPO-ptx): L_total = L_RL + γ * L_pretrain with γ ≈ 0.01–0.1. 2. Weight interpolation / model averaging: θ_final = α * θ_SFT + (1-α) * θ_RL, α ≈ 0.3–0.7; advanced variant merges SFT gradients during each RL step (Lu et al., 2024). 3. Alternating SFT ↔ DPO (LookAlike, 2025): alternate calibration-to-ground-truth and preference-learning epochs, recomputing preference data each cycle. 4. Reward robustness constraints (REWARD-ROBUST RLHF, arXiv:2409.15360): limit over-trust in weak reward signals on tasks where the RM is poor. 5. Monitor the right metrics: general-capability benchmarks, generation length distribution, KL divergence, refusal rate; trigger early stopping or LR reduction if general ability drops beyond threshold for 2 consecutive epochs.

Implications

SFT's solution is a local optimum; RL must cross a valley to reach a higher peak—the dip is the price of escaping local optima.
If RL shows *no* dip, the KL constraint may be too tight, the reward model too weak, or the learning rate too small.
Sparsity = efficiency opportunity: freeze most parameters and train only the critical 5–30%, or apply sparse-LoRA on the overlapping subnetwork.

Takeaways

1. Don't panic at the dip—give it 3–4 epochs to traverse the valley. 2. Use the right tools: pretraining mix, model averaging, alternating optimization, and robust rewards are necessities, not luxuries.

References

InstructGPT (2022): Training language models to follow instructions with human feedback
Mitigating the Alignment Tax of RLHF (ACL 2024, arXiv:2309.06256)
Reinforcement Learning Finetunes Small Subnetworks in LLMs (arXiv:2505.11711)
DPO survey (arXiv:2410.15595)
REWARD-ROBUST RLHF (arXiv:2409.15360)
RLHF in an SFT Way (arXiv:2502.11026)
RLHF — Fine-Tuning a Sparse Subnetwork in LLMs (arXiv:2507.17107, withdrawn)