English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

ZIP-SR and ZE-EDEN: Redesigning 4-bit AdamW Optimizer-State Quantization in Preconditioner Space

Forum topic · 小凯 · 2026-10-10

Summary

Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. This paper redesigns 4-bit optimizer-state quantization from the perspective of rounding space, i.e., the coordinate in which a quantizer chooses between adjacent reconstruction levels. A local analysis of the quantization cell adjacent to zero for the second moment shows that small mean state error does not imply small mean preconditioner error at the next step, and a one-dimensional quadratic construction reveals qualitatively different dynamics under state-space versus preconditioner-space rounding. These findings motivate Zero-Inclusive Preconditioner-space Stochastic Rounding (ZIP-SR), which keeps zero in the second-moment codebook and computes stochastic rounding probabilities in preconditioner space, and Zero-Excluding EDEN calibration (ZE-EDEN), which uses a zero-excluded codebook with block rescaling to reduce preconditioner distortion. Both configurations use 4-bit NF4 for the first moment and targeted stochastic rounding of the LM-head first moment over the final 10% of training. On GPT- and Llama-style pretraining from 130M to 2.7B parameters, both methods narrow the average validation-loss gap between TorchAO 4-bit AdamW and 32-bit AdamW at every model size, with a maximum reported reduction of 70%, and match near-32-bit downstream performance in full-parameter supervised fine-tuning. Source: arXiv:2610.12444.

Paper Overview

Field: ML Authors: Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter, Leonardo Neves, Aman Gupta, Hiroto Udagawa, Abhishek Shivanna, Daniel Silva, Rohan Ramanath Published: 2026-10-08 arXiv: 2610.12444

Summary

Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. This work redesigns 4-bit optimizer-state quantization for AdamW from the perspective of *rounding space*: the coordinate in which a quantizer chooses between adjacent reconstruction levels.

For the second moment, a local analysis of the quantization cell adjacent to zero shows that small mean state error need not imply small mean preconditioner error at the next step. A one-dimensional quadratic construction further shows qualitatively different optimization dynamics under state-space and preconditioner-space rounding.

These results motivate two methods:

  • ZIP-SR (Zero-Inclusive Preconditioner-space Stochastic Rounding): retains zero in the second-moment codebook and computes stochastic rounding probabilities in preconditioner space.
  • ZE-EDEN (Zero-Excluding EDEN calibration): uses a second-moment codebook that excludes zero and rescales quantized second-moment blocks to mitigate preconditioner distortion caused by a positive quantization floor.
Both configurations use 4-bit NF4 for the first moment and apply targeted stochastic rounding to the LM-head first moment over the final 10% of training.

Results

In GPT- and Llama-style pretraining experiments at 130M to 2.7B parameters, both methods narrow the average validation-loss gap between TorchAO 4-bit AdamW and 32-bit AdamW at every model size, with a maximum reported reduction of 70%. In full-parameter supervised fine-tuning, both achieve lower validation loss than TorchAO while delivering downstream task performance close to 32-bit AdamW.

Original Abstract (excerpt)

> Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. We redesign 4-bit optimizer-state quantization for AdamW from the perspective of rounding space: the coordinate in which a quantizer chooses between adjacent reconstruction levels. For the second moment, a local analysis of the quantization cell adjacent to zero shows that small mean state error need not imply small mean preconditioner error at the next step. A one-dimensional quadratic construction further shows qualitatively different optimization dynamics under state-space and preconditioner-space rounding. These results motivate Zero-Inclusive Preconditioner-space Stochastic Rounding (ZIP-SR)...

Full paper: arXiv:2610.12444

--- *Auto-collected on 2026-10-10*

Tags

#machine-learning#optimization#quantization#adamw#low-bit-training#stochastic-rounding#llm-pretraining#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178635804