Defeating Training-Inference Mismatch in RL Fine-tuning: The Case for FP16 Over BF16

Forum topic · ✨步子哥 · 2025-11-27

Summary

When fine-tuning large language models with reinforcement learning, a persistent problem known as training-inference mismatch causes training instability and collapse. Research from Sea AI Lab and the National University of Singapore (arXiv:2510.26788) traces the root cause to a commonly overlooked factor: floating-point precision. While BFloat16 (BF16) is the default for LLM training due to its wide dynamic range, its 7-bit mantissa is too imprecise, causing the inference and training engines to diverge as rounding errors accumulate across autonomic token generation. The authors show that switching the entire RL pipeline to Float16 (FP16)—which offers 8x higher precision via its 10-bit mantissa—eliminates the mismatch at its source. Experiments across dense, MoE, and LoRA models up to 14B parameters demonstrate that FP16 stabilizes training even without importance-sampling corrections (GRPO, TIS, MIS), converges faster, and closes the deployment gap. A novel 'sanity test' with mid-difficulty problems validates the approach. The work challenges the default use of BF16 in RL fine-tuning, where precision—not dynamic range—is the critical constraint.

Key points

The problem: RL fine-tuning of LLMs suffers from a training-inference mismatch. Training and inference use different execution engines (kernels, parallelism strategies), producing small numerical differences. In RL, these tiny discrepancies are amplified, causing training collapse and a deployment gap where the optimized model underperforms in its real inference environment.
Algorithmic patches fall short: Existing fixes based on importance sampling—such as GRPO-token-TIS (token-level truncated importance sampling) and GRPO-seq-MIS (sequence-level masked importance sampling)—are biased, slow to converge, or add ~25% training cost from extra forward passes. They treat symptoms: parameters are still optimized for the training engine, not the deployment engine.
The overlooked root cause: Floating-point precision. BF16's 8-bit exponent gives it FP32-like dynamic range (ideal for pretraining), but its 7-bit mantissa is coarse. During autoregressive generation, BF16 rounding errors compound token by token, so the inference policy and training policy diverge—up to 24x worse than FP16 at the sequence level, with mismatch growing exponentially with sequence length.
The fix is trivial: Simply switch the entire RL pipeline (training and inference) from BF16 to FP16. FP16's 10-bit mantissa provides 2^(10−7) = 8x finer precision. Although FP16's narrower range historically required loss scaling, fine-tuned models have well-behaved parameter and activation ranges where precision matters more than range.

| Property | FP16 | BF16 | | :--- | :--- | :--- | | Exponent bits | 5 | 8 | | Mantissa bits | 10 | 7 | | Smallest positive | ≈ 6.1 × 10⁻⁵ | ≈ 1.2 × 10⁻³⁸ | | Max value | ≈ 6.6 × 10⁴ | ≈ 3.4 × 10³⁸ | | Precision above 1 | 1 + 2⁻¹⁰ ≈ 1.000977 | 1 + 2⁻⁷ ≈ 1.007812 |

A 'sanity test' benchmark: The authors curated a dataset of math problems where the initial model scores between 20–80%—neither trivial nor impossible. Reliable algorithms should drive training accuracy toward 95–100%. Under BF16: vanilla GRPO collapsed at 73%, token-TIS collapsed at 82%, and seq-MIS converged slowly to only 95% with deployment-gap losses. Under FP16: all algorithms—including plain policy gradient with sequence-level IS—converged smoothly to 99–100%.
Broad validation: FP16 stabilized RL training for MoE models (highly sensitive due to routing), LoRA fine-tuning, and a 14B dense model, which achieved higher rewards and better AIME 2024 scores under FP16. Gains held across model families (e.g., Llama-based OctoThinker) and training frameworks.

Implications

BF16 became a default because it solved pretraining's *range* problem; RL fine-tuning's core constraint is *precision*, a different problem entirely.
With a numerically consistent foundation, biased or high-variance importance-sampling corrections become unnecessary—simple, unbiased policy gradient algorithms work robustly.
The deployment gap closes because the model is optimized in an environment numerically identical to its deployment environment.
Limitations: FP16's limited dynamic range may matter for very large models, and efficiency trends toward FP8. The paper's core value is questioning 'default settings' in the AI stack.

References

1. Qi, P., Liu, Z., Zhou, X., Pang, T., Du, C., Lee, W. S., & Lin, M. (2025). *Defeating the Training-Inference Mismatch via FP16*. arXiv:2510.26788. 2. Shao, Z., et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv:2402.03300 (GRPO). 3. Yao, Y., et al. (2025). *On the Stability of Reinforcement Learning Fine-tuning* (token-level TIS). 4. Liu, Z., et al. (2025). *Revisiting Reinforcement Learning for Language Models: On-Policy Is Not Always a Curse* (sequence-level MIS). 5. Micikevicius, P., et al. (2017). *Mixed Precision Training*.