Key Points
- Training-inference mismatch is the root cause of RL fine-tuning collapse. Sea AI Lab shows that the forward (inference) and backward (training) engines in modern RL frameworks produce numerically divergent probability distributions due to kernel and parallel-strategy differences. On DeepSeek-R1-Distill-Qwen-1.5B in BF16, this divergence grows exponentially once sequences exceed 20,000 tokens.
- BF16 is 24x worse than FP16 for RL fine-tuning. Measuring KL divergence between training-engine and inference-engine probabilities over matched rollouts, the authors report a roughly 24x mismatch under BF16 versus FP16. BF16 allocates 8 exponent bits and only 7 mantissa bits; FP16 uses 5 exponent bits and 10 mantissa bits, giving 2^10 / 2^7 = 8x finer resolution in the weight range relevant to fine-tuning.
- The Perfectible Sanity Dataset isolates algorithmic flaws. From MATH, the team selects questions where the base model scores 20-80% across 40 sampled rollouts, yielding 1,460 "golden" questions for DeepSeek-R1-Distill-Qwen-1.5B. Failure to reach 95% on this set indicates a structural algorithmic defect rather than task difficulty.
- All surveyed algorithms collapse under BF16. Vanilla GRPO tops out around 73-84% accuracy; GRPO-Token-TIS reaches 82-88% before collapsing (consistent with the bias identified by Liu et al.); GSPO stays stable longer but eventually hits a NaN gradient; only GRPO-Seq-MIS remains stable, but converges slowly to ~95% with a persistent deployment gap and 34% on AIME 2024.
- FP16 restores stability for every algorithm tested. GRPO variants, GSPO, and a plain PG-Seq-IS all train smoothly under FP16, with PG-Seq-IS reaching 39% on AIME 2024, beating the best BF16 configuration. Sequence-level importance ratios become tightly concentrated, removing the need for token-level truncation or sequence-level masking patches.
- The fix is a one-line config change. Switching
model.bfloat16()tomodel.half()plus the dynamic loss scaling already shipped in PyTorch and DeepSpeed (originally popularized in 2017) is sufficient. No additional forward passes, no ~25% overhead from importance sampling, no architecture changes. - Deployment gap disappears under FP16. Because the training and inference distributions align, models trained under FP16 deploy without the post-training accuracy drop seen under BF16.
- Generalizes to MoE and LoRA. On Qwen3-30B-A3B, FP16 yields higher training accuracy and validation reward across GRPO, GSPO, and other variants for the sparse-expert architecture, where top-k routing amplifies sensitivity to rounding error. LoRA-style low-rank updates show the same BF16-versus-FP16 gap.
- Why pre-training prefers BF16 but fine-tuning prefers FP16. Pre-training weights traverse many orders of magnitude and benefit from BF16's wide dynamic range. RL fine-tuning operates on a stabilized weight distribution where small gradient updates must be resolved precisely; mantissa resolution, not range, becomes the bottleneck.
Practical Takeaway
For RL fine-tuning pipelines, default to FP16 (with dynamic loss scaling) instead of BF16. Algorithmic patches such as token-level truncated importance sampling (Yao et al.) and sequence-level masked importance sampling (Liu et al.) are largely workarounds for a precision mismatch that FP16 eliminates directly.