This post discusses a recent paper from Ant Group's Ling team: Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe (arXiv:2606.20381). Code is not yet open-sourced.
Key points
- The symptom: A 124B MoE model trained with the NVFP4 recipe on Blackwell GPUs shows a loss curve that keeps improving but stays persistently above the BF16 baseline (~1.73% relative error), with no software-stack bug or hyperparameter issue to blame.
- The cause — Shrinkage Bias: The E2M1 FP4 format (values
{0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0}) has severely asymmetric rounding intervals at transition points. For q=2.0, the left interval is 0.25 wide but the right is 0.5, giving an expected rounding error of (left − right)/2 = −0.125 — every rounding step systematically pulls values toward zero. - Compounding decay: With quantized operands decomposed as
 = α_A·A + R_A, GEMM outputs are scaled by η ≈ α_A·α_B < 1. Over K layers,∏(k=1..K) η_k ≈ exp(-Σ δ_k)— a compound-interest formula in reverse. 1% attenuation per layer over 100 layers leaves only 37% of the signal. - The RHT paradox: The Random Hadamard Transform, a standard NVFP4 stabilizer, makes things *worse* under E2M1. RHT spreads outlier energy so dense mid-range values dominate — exactly the values landing in E2M1's most asymmetric buckets. For outlier-heavy tensors, E2M1 beats E1M2 before RHT (SQNR 21.90 vs 19.94 dB), but the ranking reverses after RHT (20.00 vs 23.19 dB). This explains why NVFP4 only dares to apply RHT on the weight-gradient (bwd_dw) path.
- The fix — UFP4: Switch to the uniformly spaced E1M2 grid (
{0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5}), which eliminates Shrinkage Bias at the source and permits RHT on all three GEMM paths (fwd_y, bwd_dx, bwd_dw). Ablations on E1M2 show loss dropping from 1.89202 (no RHT) to 1.88079 (full RHT), with stochastic rounding adding a further 0.00456 improvement. Fused RHT+quantization kernels are only 6–7% slower than quantization alone. - Chip designers: Blackwell/Rubin and MI350 treat E2M1 as the sole first-class FP4 format; this should be reconsidered. Huawei's HiFloat4 on Ascend 960, with its uniform S1P2 elements, may prove an unexpected head start.
- Framework developers: Don't blindly copy NVFP4. With uniform grids, full-path RHT is the better recipe.
- Practitioners: On E2M1-only hardware, restricting RHT to bwd_dw remains the safest choice today.
Validation results
| Model | E2M1 baseline rel. error | UFP4 rel. error | Improvement | |---|---|---|---| | Dense 1.5B | 1.2570% | 0.9673% | −23.1% | | MoE 7.9B | 2.3596% | 1.8469% | −21.7% | | MoE 124B | 1.7308% | 1.3863% | −19.9% |
Improvements hold consistently at ~20% across scales, and scaling-law fits suggest the advantage persists at larger sizes. Notably, clamping E2M1's range to simulate a uniform grid fails — the uniform grid must be native and complete (all 16 values).
Engineering implications
The deeper lesson
E2M1 became the de facto FP4 standard because it suits inference (outlier-heavy, dynamic-range-first workloads), then was carried into training by inertia. Shrinkage Bias never caused a crash — it just quietly consumed 1–2% of training quality, which at trillion-token scale is real money. The paper's sharpest insight is the RHT paradox: a widely used stabilization technique becoming an amplifier of instability under a specific format geometry. Sometimes the optimal fix is subtraction — removing a default choice that looks reasonable but is quietly toxic.
---
Paper: Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe arXiv: 2606.20381 Team: Ant Group / Ling Team