Why Low-Bit Quantization-Aware Training Is Slow: Hessian Analysis Reveals Weights Get Trapped at Saddle Points

Forum topic · 小凯 · 2026-05-19

Summary

Quantization-aware training (QAT) of language models converges extremely slowly at low bit-widths (below 4-bit). A forum post on zhichai.net discusses a Hessian spectral analysis by Li et al. showing the root cause: during QAT, weights converge to flat regions of the loss surface where most Hessian eigenvalues indicate saddle points rather than local minima. As training proceeds, eigenvalues cluster near zero and curvature approaches zero—the lower the bit-width, the flatter the surface. Gradients vanish not because a minimum is reached, but because the optimizer enters a saddle-point plateau. The authors propose WinQ, a two-stage strategy: periodic reset of weights to a linear interpolation between full-precision and quantized values (pulling weights closer to the quantization grid and re-inflating Hessian eigenvalues), and noise-injected gradients that regularize the Hessian to avoid falling back into saddle points. Across 16 configurations spanning different models, quantization methods, and bit-widths, WinQ accelerates QAT by up to 4x and improves sub-4-bit quantization accuracy by up to 8.8% at equal training cost. Open questions include choosing reset frequency and interpolation coefficients, adaptively tuning noise magnitude, and overhead on 30B+ parameter models.

Quantization-aware training (QAT) converges extremely slowly at low bit-widths (below 4-bit). It is not training instability—the loss simply stops decreasing, sitting flat like still water. Li, Liu, Yi, Zhang, Zhao, Krishnamoorthi, Khaitan, Zhang, and Li used Hessian spectral analysis to find out why.

Diagnosis: Saddle Points, Not Minima

The authors estimate the eigenvalue spectrum of the loss surface's Hessian matrix during training. In QAT, the weights converge to a flat region where most Hessian eigen directions simultaneously have positive and negative curvature—i.e., a saddle point, not a local minimum. As training continues, more and more eigenvalues cluster near zero and the curvature of the surface approaches zero. The lower the bit-width, the smaller the eigenvalue magnitudes and the flatter the surface.

This explains why QAT is slow: gradients approach zero not because the model reached the lowest point, but because it entered the flat plateau of a saddle point.

WinQ: A Two-Stage Strategy to Escape

WinQ breaks the deadlock with two techniques:

1. Periodic reset: during training, weights are periodically reset to a linear interpolation between full-precision and quantized weights. This reduces the distance to the quantization grid and forces Hessian eigenvalues to grow again. 2. Noise-injected gradients: noise is added to weights during gradient computation, regularizing the Hessian and preventing the model from falling back into saddle points.

Results

Experiments across 16 configurations (different models, quantization methods, and bit-widths) show that WinQ:

Accelerates QAT by up to 4x
Improves sub-4-bit quantization accuracy by up to 8.8% at equal training cost

Open Questions

How to choose the reset period and interpolation coefficients—too frequent destroys learned information, too infrequent fails to escape saddle points?
How can the noise injection magnitude automatically adapt to different quantization configurations?
What are the extra memory and compute costs of WinQ when training large models (30B+)?

References

1. Li, D., Liu, Z., Yi, K., et al. (2026). *WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points*. arXiv:2605.17471 [cs.LG]. 2. Nagel, M., et al. (2021). *A White Paper on Neural Network Quantization*. arXiv. 3. Dauphin, Y., et al. (2014). *Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization*. NeurIPS.