English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts

Forum topic · 小凯 · 2026-11-10

Summary

Researchers Chang Lu and Mengjiao Zhang (arXiv:2611.12832) show that FP4 pretraining slows the acquisition of rare facts compared to BF16, even when validation loss appears nearly identical. Training Transformers from scratch on synthetic biographies with simulated 4-bit matrix multiplication, they find NVFP4 requires 1.32x and 1.52x more exposures to match BF16's rare-fact recall while validation loss differs by only 0.5% and 2%. Gradient noise analysis shows noise injected into updates reduces signal-to-noise ratio and accounts for about two-thirds of the cost persisting through training, while noise in Adam's second moment behaves like a lower learning rate and fades. Simply raising the learning rate cannot recover the cost. Gradient statistics from BF16 checkpoints can rank the costs of different formats without fitting (Spearman 0.96). The authors recommend measuring rare-fact acquisition alongside loss when evaluating low-precision training recipes.

What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts

  • Field: Machine Learning
  • Authors: Chang Lu, Mengjiao Zhang
  • Published: 2026-11-09
  • arXiv: 2611.12832
  • Overview

    Low-precision pretraining is typically judged by validation loss. This paper asks a sharper question: does loss-matched training also mean matched efficiency in learning rare facts? The answer is no — FP4 pretraining systematically slows the acquisition of rare facts relative to BF16.

    Key Findings

  • Experimental setup: Transformers trained from scratch on synthetic biographies with controlled exposure counts, using simulated 4-bit matrix multiplication, in both isolated settings and mixed with web text.
  • Recall gap: NVFP4 requires 1.32x and 1.52x more exposures to match BF16's recall, while validation loss differs by only 0.5% and 2%. When both are evaluated with BF16 matrix multiplication, the gaps are 1.26x and 1.30x. At the main readout, the exposure cost at matched loss is roughly 1.2x.
  • Gradient noise analysis: Zero-mean gradient noise reproduces the cost of quantized backpropagation, which itself barely moves loss. In controlled experiments with twice the NVFP4 error:
  • Noise injected into Adam's second moment behaves like a lower learning rate, and its cost fades over training.
  • Noise injected into updates lowers the signal-to-noise ratio and carries about two-thirds of the cost through the end of training.
  • Learning rate cannot compensate: Raising the learning rate to offset smaller average step sizes fails to recover the cost, for both this noise and NVFP4 itself.
  • Predictive metric: Gradient statistics from BF16 checkpoints rank the costs of formats, partial quantization, and noise controls without any fitting (Spearman correlation 0.96).

Recommendation

When evaluating low-precision training recipes, practitioners should measure rare-fact acquisition alongside validation loss, since loss curves alone miss a meaningful capability gap.

Original Abstract

> We show that FP4 pretraining slows the acquisition of rare facts compared to BF16, requiring 1.2-1.5x more exposures to match recall while validation loss differs by only 0.5-2%. We recommend measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.

Tags

#machine-learning#fp4#quantization#pretraining#bf16#rare-facts#gradient-noise

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637026