What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
- Field: Machine Learning
- Authors: Chang Lu, Mengjiao Zhang
- Published: 2026-11-09
- arXiv: 2611.12832
- Experimental setup: Transformers trained from scratch on synthetic biographies with controlled exposure counts, using simulated 4-bit matrix multiplication, in both isolated settings and mixed with web text.
- Recall gap: NVFP4 requires 1.32x and 1.52x more exposures to match BF16's recall, while validation loss differs by only 0.5% and 2%. When both are evaluated with BF16 matrix multiplication, the gaps are 1.26x and 1.30x. At the main readout, the exposure cost at matched loss is roughly 1.2x.
- Gradient noise analysis: Zero-mean gradient noise reproduces the cost of quantized backpropagation, which itself barely moves loss. In controlled experiments with twice the NVFP4 error:
- Noise injected into Adam's second moment behaves like a lower learning rate, and its cost fades over training.
- Noise injected into updates lowers the signal-to-noise ratio and carries about two-thirds of the cost through the end of training.
- Learning rate cannot compensate: Raising the learning rate to offset smaller average step sizes fails to recover the cost, for both this noise and NVFP4 itself.
- Predictive metric: Gradient statistics from BF16 checkpoints rank the costs of formats, partial quantization, and noise controls without any fitting (Spearman correlation 0.96).
Overview
Low-precision pretraining is typically judged by validation loss. This paper asks a sharper question: does loss-matched training also mean matched efficiency in learning rare facts? The answer is no — FP4 pretraining systematically slows the acquisition of rare facts relative to BF16.
Key Findings
Recommendation
When evaluating low-precision training recipes, practitioners should measure rare-fact acquisition alongside validation loss, since loss curves alone miss a meaningful capability gap.
Original Abstract
> We show that FP4 pretraining slows the acquisition of rare facts compared to BF16, requiring 1.2-1.5x more exposures to match recall while validation loss differs by only 0.5-2%. We recommend measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.