> *GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling* > arXiv: 2604.18556 | 2026
---
A Deployer's Dilemma
Suppose you want to run a 70B-parameter LLM locally. Your GPU has 24GB of VRAM.
At FP16, a 70B model needs ~140GB. 4-bit quantization compresses that to ~35GB — barely fits on two cards. 2-bit quantization gets it down to ~17.5GB — one card suffices.
The problem: the more you compress, the dumber the model gets. Classical scalar quantization methods (GPTQ, AWQ) hold up well at 3-4 bits but degrade sharply at 2-3 bits.
A "second generation" of methods — vector/grid quantization (AQLM, QTIP) — performs better at low bit widths, but is complex to implement and costly to deploy.
This paper asks a direct question: is the accuracy gap between scalar and vector quantization fundamental, or just a matter of insufficient optimization?
The answer: it's just insufficient optimization.
---
GSQ: Bridging Discrete and Continuous with Gumbel-Softmax
GSQ (Gumbel-Softmax Quantization) has an elegant core idea.
Quantization is inherently a discrete optimization problem: each weight must be assigned to one of a finite set of values (e.g., only 4 possible values for 2-bit quantization). Discrete optimization is hard — you can't take gradients through discrete variables.
The traditional approach: first determine which discrete value each weight maps to (via some heuristic), then only optimize the discrete values themselves. This "two-step" strategy is too rigid at very low bit widths.
GSQ instead uses the Gumbel-Softmax relaxation, a technique that turns discrete choices into continuous probability distributions while preserving differentiability.
Concretely, GSQ jointly optimizes: 1. Which quantization level each weight is assigned to (via Gumbel-Softmax soft selection) 2. Per-group scaling factors (continuous values calibrating the quantized value range)
The key innovation: GSQ makes the relaxation dimension exactly match the number of levels at the target bit width. For 2-bit quantization with 4 levels, the Gumbel-Softmax outputs only 4 probabilities. This makes the relaxation tighter and the optimization more precise.
---
Results: Scalar Quantization Catches Up
Experiments on Llama-3.1-8B and 70B-Instruct are impressive:
- In the 2-3 bit (bpp) range, GSQ essentially eliminates the accuracy gap between scalar and vector quantization
- On standard benchmarks, GSQ matches or even exceeds complex vector quantization methods like AQLM
- As a scalar method, GSQ is far simpler to implement and deploy
---
Why This Matters
From a practical standpoint, GSQ's value is substantial:
1. Simpler deployment: scalar quantization directly replaces model weights, requiring no inference engine changes 2. Faster inference: scalar operations are more efficient and achieve higher hardware utilization than vector operations 3. Broader coverage: nearly all inference frameworks support scalar quantization, while vector quantization support is spotty
As the paper puts it: "the gap between scalar and vector quantization is more an optimization gap than a fundamental limitation."
---
My Takeaways
This paper suggests a broader pattern: many problems that seem to require more complex methods are actually just simple methods that weren't optimized well enough.
In quantization, the conventional wisdom was "scalar quantization has hit its ceiling; you need vector quantization to break through." GSQ shows that conclusion came too early — scalar quantization's ceiling is far from reached; it just needed a better optimization strategy.
The use of Gumbel-Softmax is also clever. Originally designed for discrete variational autoencoders, it's now applied to quantization. Good mathematical tools often have broad applicability — the key is spotting the "bridge."
---
Paper: arxiv.org/abs/2604.18556