ScaleSearch: Stop Using Max Values for Block Floating Point Scales

Forum topic · Anonymous · 2026-05-18

Summary

Block floating point quantization typically uses a block's maximum absolute value as the scaling factor, but Gupta, Prairie, Wu and collaborators (including Tri Dao and Chris De Sa) show this common heuristic is suboptimal. The max-value scale prevents overflow but leaves most values squeezed into the bottom of the low-precision range, wasting quantization steps. ScaleSearch instead searches for the optimal scale over the discrete grid defined by the mantissa bits of the target format. For NVIDIA's NVFP4 4-bit format (Blackwell architecture), the searched scale reduces quantization error by 27% compared to the max-value method. ScaleSearch composes with existing post-training quantization: on Qwen3-8B, PTQ quality on MATH500 improves by up to 15 points. The authors also propose ScaleSearchAttention, an NVFP4-based low-precision attention implementation that reaches within 0.77 perplexity of Llama 3.1 70B baselines. Open questions include the search's latency overhead and optimality guarantees. Source: arXiv:2605.12464.

The standard recipe for neural network quantization is to normalize a group of numbers (a block) and use a single shared scaling factor to compress their range into what a low-precision format can represent. How should this scale be chosen? Almost everyone uses the same method: find the maximum absolute value in the block and set the scale to it.

Gupta, Prairie, Wu and their team (including Tri Dao and Chris De Sa) found a fact the industry has overlooked: the maximum value is not necessarily optimal.

The intuition: when you scale by the block's maximum, you guarantee no value overflows the representable range of the low-precision format. But the cost? Most values in the block get compressed into the bottom region of the format, wasting a large portion of the available quantization steps—like cramming everyone onto one floor of a building while the nine floors above sit empty.

What ScaleSearch does

ScaleSearch searches for the optimal scale instead of taking the maximum. The search space is the discrete grid defined by the mantissa bits of the low-precision format. For NVFP4—NVIDIA's latest 4-bit floating point format for the Blackwell architecture—the scale found by searching reduces quantization error by 27% compared to the max-value method.

Results

Post-training quantization: ScaleSearch can be combined with existing PTQ methods. For the Qwen3-8B model, PTQ quality on the MATH500 benchmark improves by up to 15 points.
Attention: For attention computation, critical in LLM inference, the team introduces ScaleSearchAttention, an NVFP4-based low-precision attention implementation that achieves a perplexity within 0.77 of Llama 3.1 70B—essentially lossless.

Open questions

Search overhead: does inserting a search step into the inference path affect latency? The paper calls it "micro-search," but the exact number of search steps and time cost are not quantified.
Optimality guarantees: does the search find a local or global optimum? What is the stopping criterion?

---

References

1. Gupta, T., Prairie, H., Wu, X., et al. (2026). *Search Your Block Floating Point Scales!* arXiv:2605.12464 [cs.LG]. 2. Rouhani, B., et al. (2023). *Microscaling Data Formats for Deep Learning*. arXiv:2310.10537. 3. NVIDIA. (2025). *NVFP4: 4-bit Floating Point for Blackwell Architecture*. NVIDIA Developer Documentation.