ScaleSearch: Tri Dao Team Shows Searching Beats Default Max in Quantization Scales

Forum topic · 二一 · 2026-05-13

Summary

A new paper from Tri Dao's team (authors of FlashAttention and Mamba), ScaleSearch, challenges the standard practice in low-precision GPU computing where a block's quantization scale factor is set to the maximum absolute value in the block. The authors prove this default is suboptimal and instead search for the optimal scale, exploiting mantissa bits in microscaling formats such as NVFP4 to minimize quantization error. ScaleSearch integrates seamlessly into existing post-training quantization (PTQ) and low-precision attention methods. Reported results include a 27% reduction in quantization error for NVFP4, a 15-point PTQ improvement for Qwen3-8B on MATH500, up to 0.77 perplexity improvement for Llama 3.1 70B on Wikitext-2, and ScaleSearchAttention achieving near-zero loss for NVFP4 attention. The core insight: hardware default parameters are often not optimal, and simply searching rather than taking defaults yields meaningful gains.

Tri Dao's team (authors of FlashAttention and Mamba) released a new paper addressing a simple but important question in low-precision GPU computing: how should a scale factor be assigned to a data block when computing in formats like NVFP4?

The standard approach uses the maximum absolute value within the block as the scale factor. The paper proves this default is suboptimal.

Key Innovation: ScaleSearch

ScaleSearch searches for the optimal scale factor instead of using the default maximum. It exploits the mantissa bits in microscaling formats to minimize quantization error, and integrates seamlessly into existing PTQ (post-training quantization) and low-precision attention methods.

Key Results

Quantization error reduced by 27% (NVFP4)
Qwen3-8B PTQ performance on MATH500 improved by 15 points
Llama 3.1 70B Wikitext-2 PPL improved by up to 0.77
ScaleSearchAttention achieves near-zero loss for NVFP4 attention

Core Insight

Hardware default parameters are often not optimal — as long as you are willing to *search* rather than *take the default*, you can squeeze out substantial performance gains.

Reference: ScaleSearch, arXiv:2605.12464, Tri Dao, Daniel Y. Fu et al.