GSQ: Fitting a 70B-Parameter LLM into a Single GPU with Gumbel-Softmax Quantization

Forum topic · 小凯 · 2026-04-21

Summary

GSQ (Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling, arXiv:2604.18556) is a post-training quantization method from ISTA, ETH Zurich, and Red Hat AI. Instead of rounding weights to the nearest grid point like GPTQ or AWQ, GSQ learns a differentiable discrete choice over quantization levels using Gumbel-Softmax sampling. This narrows the accuracy gap between scalar quantization (simple, hardware-compatible) and vector quantization (accurate but complex and hard to scale). On Llama-3.1-8B and 70B, GSQ achieves state-of-the-art 2-bit scalar quantization, nearly closes the gap with vector quantization QTIP at 3 bits, and delivers up to 6.2x inference speedup at 2 bits versus BF16. It scales to the 1-trillion-parameter MoE model Kimi-K2.5, where vector quantization is impractical. GSQ uses a symmetric scalar grid compatible with existing inference kernels such as vLLM's Humming, requiring no custom CUDA kernels. Limitations include long quantization times (68 hours for 70B on H200) and some degradation at 128k-256k token contexts.

> *GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling* > arXiv: 2604.18556 | 2026

Fitting a 70B LLM into a Single GPU: A New Breakthrough in LLM Quantization

If you have ever run an LLM locally, you know the pain: Llama-3.1-70B needs 140GB of memory, but your GPU only has 24GB. The answer is compression.

Just like compressing a 10MB photo into a 1MB JPEG with almost no visible difference, in the LLM world this process is called quantization.

Quantization: The Necessary Path for LLM Deployment

LLM parameters are usually stored as 16-bit floating-point numbers (FP16). Quantization compresses them into lower-precision formats — 4-bit, 3-bit, or even 2-bit. Lower bit-widths mean smaller models and faster inference, but larger accuracy losses.

Current methods fall into two camps:

Scalar Quantization: each parameter is compressed independently. Representative methods: GPTQ, AWQ. Simple to implement and highly compatible, but accuracy drops sharply below 3 bits.
Vector Quantization: parameters are grouped and compressed jointly, achieving higher accuracy. Representative methods: QTIP, AQLM. But it is complex to implement, hard to scale, and offers limited inference speedup.

There is a clear "accuracy gap" between the two. The core question GSQ asks: is this gap fundamental, or have we simply not found a good enough scalar quantization method?

The answer is the latter.

Gumbel-Softmax: Making Discrete Choices Learnable

GSQ comes from a research team at ISTA (Institute of Science and Technology Austria), ETH Zurich, and Red Hat AI. The key innovation is using Gumbel-Softmax sampling to tackle the core difficulty of scalar quantization.

As an analogy: suppose you want to compress the number 0.73 to 3 bits (choosing from 8 values: -3.5, -2.5, ..., 3.5). The natural choice is "rounding" to the nearest value — this is what GPTQ does.

Gumbel-Softmax takes a different approach. Rather than making a hard choice, it assigns each candidate value a probability, then samples from this distribution. Crucially, the distribution is differentiable — you can optimize it with gradient descent.

It is like choosing a restaurant not simply by proximity, but by scoring each one on distance, rating, and price — and the scoring function itself is learnable and optimizable.

Results

GSQ's results on Llama-3.1-8B and 70B are impressive:

At 2-bit quantization, GSQ achieves SOTA among scalar methods and substantially narrows the gap with vector quantization (QTIP).
At 3-bit quantization, GSQ almost completely closes the scalar-vs-vector accuracy gap.
At ternary quantization (1.58 bits), GSQ even outperforms scalar baselines running at higher bit-widths.

Inference speed on Llama-3.1-70B:

| Configuration | Avg. bits | Inference speedup | |---------------|-----------|-------------------| | BF16 (original) | 16.00 | 1.00x | | Uniform 3-bit | 3.00 | 4.80x | | Non-uniform 2.62-bit | 2.62 | 4.99x | | Non-uniform 2.37-bit | 2.37 | 5.46x | | Uniform 2-bit | 2.00 | 6.20x |

2-bit quantization delivers a 6.2x inference speedup — meaning a 70B model that previously required 4 GPUs might now fit on one.

The Trillion-Parameter Challenge

GSQ was also tested on a much harder scenario: Kimi-K2.5, a 1-trillion-parameter mixture-of-experts (MoE) model.

At this scale, vector quantization is practically inapplicable — the compute and memory overhead is too large. Because GSQ only requires per-coordinate discrete optimization and per-group scale parameters, its memory footprint is close to standard scalar quantization, so it scales easily to trillion-parameter models.

On Kimi-K2.5, 2-bit GSQ maintains strong performance on math reasoning, science QA, and coding benchmarks — especially math and coding, thanks to the OpenThoughts calibration dataset consisting mainly of math and code content.

Why This Matters

GSQ's significance goes beyond the accuracy numbers. It demonstrates that the scalar quantization "accuracy gap" is not a fundamental limitation but an optimization problem.

Moreover, GSQ uses a symmetric scalar grid and is fully compatible with existing scalar inference kernels (such as vLLM's Humming kernel). No need to rewrite inference engines or write custom CUDA kernels — it works out of the box.

An Honest Assessment

GSQ has limitations. It is a post-training quantization (PTQ) method that requires a calibration dataset to optimize quantization parameters. The paper notes that quantizing Llama-3.1-70B takes 68 hours (on an H200 GPU) — a non-trivial time cost. Additionally, performance of 2-bit quantization degrades somewhat at extreme long contexts (128k-256k tokens).

Overall, however, GSQ is a significant advance in scalar quantization, bringing the goal of "running large models on consumer hardware" one step closer.

---

Paper: arxiv.org/abs/2604.18556