*GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling* arXiv: arxiv.org/abs/2604.18556 | 2026
---
The Deployer's Dilemma
Suppose you want to run a 70B-parameter LLM locally, but your GPU has only 24GB of VRAM. At FP16, a 70B model needs 140GB — nearly 6x more than you have.
The answer is quantization: compressing each parameter from 16 bits down to fewer. 4-bit quantization shrinks 140GB to 35GB (two GPUs), while 2-bit brings it to 17.5GB — a single card suffices.
The catch: the more you compress, the dumber the model gets. Traditional scalar quantization methods (GPTQ, AWQ) hold up reasonably well at 3-4 bits but degrade sharply at 2-3 bits.
This gave rise to "second-generation" methods — vector/grid quantization (AQLM, QTIP). They perform better at low bits, but are complex to implement and hard to deploy.
This paper asks a direct question: is the accuracy gap between scalar and vector quantization fundamental, or just a matter of suboptimal optimization?
The answer: it's just suboptimal optimization.
---
GSQ: Bridging Discrete and Continuous with Gumbel-Softmax
The core idea of GSQ (Gumbel-Softmax Quantization) is elegant.
Quantization is inherently a discrete optimization problem: each weight must be assigned to one of a finite set of discrete values (e.g., 2-bit quantization offers only 4 possible values). Discrete optimization is hard — you can't take gradients through discrete variables.
The conventional approach is two-step: first decide which discrete value each weight maps to (via some heuristic), then optimize only the discrete levels themselves. This strategy isn't flexible enough at very low bit widths.
GSQ uses the Gumbel-Softmax relaxation to solve this. Gumbel-Softmax turns discrete choices into continuous probabilities — letting you "softly" pick a discrete value while keeping everything differentiable.
Concretely, GSQ jointly optimizes: 1. Which quantization level each weight is assigned to (via Gumbel-Softmax soft selection) 2. Per-group scaling factors (continuous values that calibrate the quantized numeric range)
The key innovation: GSQ strictly matches the relaxation's dimensionality to the number of levels at the target bit width. For 2-bit quantization with 4 levels, the Gumbel-Softmax outputs exactly 4 probabilities. This makes the relaxation tighter and the optimization more precise.
---
Results: Scalar Quantization Catches Up with Vector Quantization
Experiments on Llama-3.1-8B and 70B-Instruct are impressive:
- In the 2-3 bit (bpp) regime, GSQ essentially eliminates the accuracy gap between scalar and vector quantization
- On standard benchmarks, GSQ performs close to or better than complex vector quantization methods like AQLM
- As a scalar method, GSQ is far simpler to implement and deploy
---
Why This Matters
From a practical standpoint, GSQ's value is substantial:
1. Simpler deployment: scalar quantization directly replaces model weights — no inference engine modifications needed 2. Faster inference: scalar operations are more hardware-efficient than vector operations 3. Broader coverage: nearly all inference frameworks support scalar quantization; vector quantization support is patchy
As the paper puts it: the gap between scalar and vector quantization is "more an optimization gap than a fundamental limitation."
---
My Take
This paper points to a broader pattern: many problems that seem to require more complex methods are actually just simple methods that weren't optimized well.
In the quantization field, many believed "scalar quantization has hit its ceiling; you need vector quantization to break through." GSQ shows that conclusion came too early — the ceiling of scalar quantization is far from reached; it just needed a better optimization strategy.
The use of Gumbel-Softmax is also clever. Originally designed for discrete variational autoencoders, it now tackles quantization. Good mathematical tools often have broad applicability — the key is spotting the bridge.
---
Paper: arxiv.org/abs/2604.18556