Defeating the Training-Inference Mismatch in RL Fine-tuning: Why FP16 Beats BF16

Forum topic · ✨步子哥 · 2025-11-27

Summary

This forum post reviews a research paper from Sea AI Lab and the National University of Singapore showing that the notorious train-inference mismatch in reinforcement learning fine-tuning of large language models stems primarily from the BF16 floating-point format rather than RL algorithms. BF16's 7-bit mantissa makes it roughly 8x less precise than FP16, so tiny rounding differences between the rollout and training engines snowball across autoregressive token generation, destabilizing GRPO-style training and creating a deployment gap. Simply switching the entire RL pipeline from BF16 to FP16 (with loss scaling) aligns the two numerical engines, eliminating the need for costly and biased importance-sampling corrections like TIS and MIS. In 'sanity tests' and experiments on MoE models, LoRA fine-tuning, and a 14B dense model, FP16 yields stable convergence, faster training, and higher scores on AIME 2024. The takeaway: revisit default precision choices—precision matters more than dynamic range during RL fine-tuning.

When AI Loses Itself Between Training and Inference: The FP16 Discovery

*An in-depth walkthrough of the paper "Defeating the Training-Inference Mismatch via FP16" (Qi et al., 2025, arXiv:2510.26788), which traces RL fine-tuning instability to floating-point precision rather than algorithm design.*

The Prologue: An Unruly Beast and a Fragile Rein

Fine-tuning large language models (LLMs) with reinforcement learning (RL) is often described as taming a beast of near-infinite intelligence. The process is fragile: models frequently suffer catastrophic "training collapse." For years, researchers have been haunted by the train-inference mismatch. For efficiency, we run two different "engines": one performs fast rollouts during training (inference), the other updates gradients (training). In theory they should behave identically like digital twins—but due to hardware and kernel differences, identical computations produce slightly different numerical results.

Under RL's sensitive amplification, these tiny discrepancies explode into a double catastrophe: the training signal becomes unreliable (like navigating with a distorted map), and even a successfully trained model is optimized for the wrong engine, leaving a "deployment gap" at inference time.

The Craftsmen's Patch: Algorithmic Fixes

The standard remedy builds on importance sampling: if the inference policy is μ and the training policy is π, reweight samples to correct the distributional gap. Notable variants include:

GRPO-token-TIS (token-level Truncated Importance Sampling): corrects at each token, but the truncation itself introduces bias—training still eventually collapses.
GRPO-seq-MIS (sequence-level Masked Importance Sampling): corrects only when the whole sequence's deviation is bounded—more stable, but slow to converge.

Both approaches share two fatal flaws:

1. Cost: they require an extra forward pass per update, adding roughly 25% training overhead—significant for multi-million-dollar LLM training. 2. Treating symptoms, not the cause: parameters are still optimized under π's numerical view, so the deployment gap with μ persists.

The Root Cause: A Precision Crisis

Researchers at Sea AI Lab and NUS asked a disruptive question: what if the problem isn't the algorithm, but the numerical ground we stand on—floating-point precision?

BFloat16 (BF16) has become the default for large-model training thanks to its 8-bit exponent, giving FP32-like dynamic range. But it allocates only 7 bits to the mantissa. During autoregressive generation, per-token rounding differences—caused by divergent CUDA kernels and parallelization strategies—snowball across thousands of steps, like repeatedly photocopying a photo until it's unrecognizable.

FP16 vs BF16: A Fated Duel

Float16 (FP16) allocates only 5 bits to the exponent but 10 bits to the mantissa, making it $2^{(10-7)} = 8$ times more precise above 1:

| Property | FP16 (IEEE 754) | BF16 (Brain Float) | | :--- | :--- | :--- | | Exponent bits | 5 | 8 | | Mantissa bits | 10 | 7 | | Smallest positive | ≈ 6.1 × 10⁻⁵ | ≈ 1.2 × 10⁻³⁸ | | Max value | ≈ 6.6 × 10⁴ | ≈ 3.4 × 10³⁸ | | Distinguishable precision >1 | 1 + 2⁻¹⁰ ≈ 1.000977 | 1 + 2⁻⁷ ≈ 1.007812 |

> Note: Floating-point numbers represent reals via sign, exponent, and mantissa. The exponent determines range; the mantissa determines precision. BF16 and FP16 spend the same 16-bit budget differently.

During RL fine-tuning, the model's parameter and activation ranges have already been tamed by pretraining—dynamic range is no longer the bottleneck; precision is. FP16's fine granularity acts as a buffer, absorbing implementation-level rounding differences. In the paper's Figure 2, BF16 shows the inference policy μ and training policy π scattered far from the identity line, while FP16 snaps them tightly onto it. At the sequence level, BF16 mismatch grows exponentially with text length; FP16 keeps it near zero—a gap of up to 24x.

The Fix: A Few Lines of Configuration

The intervention is remarkably simple: switch the entire RL pipeline—both training and inference—from BF16 to FP16 (with standard loss scaling). No new algorithms, no elaborate engineering alignment.

Consequences:

1. The age of expensive algorithmic patches ends. With the twin engines numerically aligned, unbiased classical policy gradients with importance weights work again, and the deployment gap closes—models optimized in training are optimal in deployment. 2. Unprecedented stability and efficiency. Under BF16, GRPO, TIS, and MIS all collapse or underperform; under FP16, training curves become flat and monotonic, converging faster with much higher final performance.

The Sanity Test: A Litmus Trial for RL Algorithms

The authors designed a "sanity test": from a large math problem bank, they filtered problems where the initial model's accuracy is between 20% and 80%—achievable but not trivial. A reliable algorithm should drive training accuracy toward 95–100% on this set.

Results were one-sided:

BF16 camp: vanilla GRPO collapsed early (max 73%); GRPO-token-TIS collapsed at 82%; GRPO-seq-MIS stayed stable but crawled to 95% with visible deployment-gap losses on eval.
FP16 camp: every algorithm—including the plainest PG-Seq-IS—surged smoothly to 99–100%. Sequence-level importance sampling, nearly unusable under BF16, became stable and efficient.

FP16's Widespread Victories

MoE models: notoriously numerically sensitive due to expert routing; FP16 turned wobbly training into bedrock stability across GRPO-Seq-MIS, GRPO-Token-TIS, and PG-Seq-TIS.
LoRA fine-tuning: BF16-based RL collapses after a few hundred steps; FP16 trains smoothly to the end.
14B dense model: FP16's reward curves rise much faster and achieve higher scores on AIME 2024.

Generality holds across model families (e.g., Llama-based OctoThinker) and training frameworks.

Reflections: Rethinking Our Contract with Numbers

BF16 won because it solved pretraining's "range anxiety," and we carried that habit into every stage—including RL fine-tuning. But RL's core tension has shifted from range to precision. On BF16's imprecise foundation, bias-correcting algorithms either collapse (GRPO) or pay in variance and slow convergence (MIS); FP16 raises the foundation itself, reducing both bias and variance simultaneously.

FP16 is not a universal answer—very large models may still strain its limited range, and efficiency trends may push toward FP8. But the paper's real value is breaking dogma: question the "default settings." Sometimes the most elegant solution hides in the most overlooked corner. Returning to FP16 is not regression but a spiral ascent—a wiser, more precise choice made after understanding the problem's essence.

References

1. Qi, P., Liu, Z., Zhou, X., Pang, T., Du, C., Lee, W. S., & Lin, M. (2025). *Defeating the Training-Inference Mismatch via FP16*. arXiv preprint arXiv:2510.26788. 2. Shao, Z., et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv preprint arXiv:2402.03300. (Original GRPO paper) 3. Yao, Y., et al. (2025). *On the Stability of Reinforcement Learning Fine-tuning*. (Token-level TIS correction) 4. Liu, Z., et al. (2025). *Revisiting Reinforcement Learning for Language Models: On-Policy Is Not Always a Curse*. (Sequence-level MIS correction) 5. Micikevicius, P., et al. (2017). *Mixed Precision Training*. (Classic FP16 + loss scaling paper)