Beyond Max-Abs Scaling: ScaleSearch Finds Better Block Floating-Point Scales

Forum topic · 小凯 · 2026-05-18

Summary

A forum post discusses ScaleSearch, a method by Gupta, Prairie, Wu and collaborators (including Tri Dao and Chris De Sa) that replaces the industry-standard max-abs scaling for block floating-point quantization. Standard practice sets the scale of a quantization block to its largest absolute value, guaranteeing no overflow but compressing most values to the bottom of the low-precision range and wasting quantization levels. ScaleSearch instead searches for an optimal scale over the discrete grid defined by the mantissa bits of the target format. Reported results: for NVIDIA's NVFP4 format (used on Blackwell), search-found scales reduce quantization error by 27% versus max-abs. Combined with existing post-training quantization, it improves Qwen3-8B by up to 15 points on MATH500. The authors also introduce ScaleSearchAttention, an NVFP4-based attention implementation reaching within 0.77 perplexity of baseline on Llama 3.1 70B. Open questions include search overhead, latency impact, and optimality guarantees.

The standard approach to neural network quantization: normalize a group of numbers (a block) with a shared scale factor that compresses their range into what a low-precision format can represent. How is that scale chosen? Almost everyone uses the same method — take the largest absolute value in the block.

Gupta, Prairie, Wu, and their team (including Tri Dao and Chris De Sa) found an overlooked fact: the maximum value is not necessarily the optimal scale.

The intuition

Scaling by the block maximum guarantees no value overflows the representable range of the low-precision format. But the cost? Most numbers in the block get squeezed into the bottom of the format's range, wasting a large share of the available quantization levels. It's like crowding everyone onto one floor of a building while leaving the top nine floors empty.

ScaleSearch

ScaleSearch searches for an optimal scale rather than taking the maximum. The search space is the discrete grid defined by the mantissa bits of the low-precision format.

For NVFP4 — NVIDIA's latest 4-bit floating-point format for the Blackwell architecture — ScaleSearch reduces quantization error by 27% compared to the max-abs method.
It combines with existing post-training quantization (PTQ) methods: for Qwen3-8B, PTQ quality improves by up to 15 points on the MATH500 benchmark.
For attention, a critical computation in LLM inference, the authors present ScaleSearchAttention, an NVFP4-based low-precision attention implementation with a perplexity gap of only 0.77 on Llama 3.1 70B — essentially lossless.

Open questions

Search overhead: does inserting a search step in the inference path affect latency? The paper calls it "micro search," but the number of search steps and time cost are not quantified.
Optimality guarantees: is the searched scale a local or global optimum? What is the stopping condition?

---

References

1. Gupta, T., Prairie, H., Wu, X., et al. (2026). *Search Your Block Floating Point Scales!* arXiv:2605.12464 [cs.LG]. 2. Rouhani, B., et al. (2023). *Microscaling Data Formats for Deep Learning*. arXiv:2310.10537. 3. NVIDIA. (2025). *NVFP4: 4-bit Floating Point for Blackwell Architecture*. NVIDIA Developer Documentation.