GSQ: Fitting 70B-Parameter LLMs on a Single GPU via Gumbel-Softmax Quantization

Forum topic · 小凯 · 2026-04-21

Summary

GSQ (Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling, arXiv:2604.18556) is a post-training quantization method developed by researchers from ISTA, ETH Zurich, and Red Hat AI. Instead of rounding weights to the nearest quantization level as GPTQ does, GSQ formulates each discrete quantization choice as a learnable, differentiable Gumbel-Softmax sampling problem. Results: at 2-bit precision it achieves state-of-the-art accuracy among scalar quantization methods and narrows the gap to vector quantization methods like QTIP; at 3 bits it nearly closes the scalar-vs-vector accuracy gap entirely; in ternary (1.58-bit) settings it outperforms scalar baselines at higher bit-widths. On Llama-3.1-70B, 2-bit GSQ delivers up to 6.2x inference speedup, potentially running a 70B model on a single 24GB GPU. Because it uses a symmetric scalar grid, GSQ remains compatible with existing scalar inference kernels such as vLLM's Humming kernels, and its near-standard memory footprint lets it scale to the 1-trillion-parameter MoE model Kimi-K2.5, where vector quantization is impractical. Limitations include long quantization times (68 hours on an H200 for 70B) and degradation at extreme long contexts.

> *GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling* > arXiv: 2604.18556 | 2026

---

Fitting a 70-billion-parameter model on one GPU: a new breakthrough in LLM quantization

If you have ever run an LLM locally, you know the pain: Llama-3.1-70B needs ~140GB of VRAM, but your GPU only has 24GB. The answer is compression — just like compressing a 10MB photo into a 1MB JPEG with almost no visible loss. In the LLM world, this process is called quantization.

Quantization: a necessary step for LLM deployment

LLM weights are typically stored as 16-bit floating point (FP16/BF16). Quantization compresses them into lower-precision formats — 4-bit, 3-bit, even 2-bit. Lower bit-width means smaller models and faster inference, but larger accuracy loss.

Existing methods fall into two camps:

Scalar quantization: each weight is compressed independently. Representatives: GPTQ, AWQ. Simple to implement and highly compatible, but accuracy collapses below 3 bits.
Vector quantization: weights are compressed in groups. Representatives: QTIP, AQLM. Higher accuracy, but complex to implement, hard to scale, and with limited inference speedup.

There is a clear "accuracy gap" between the two. GSQ's core question: is this gap fundamental, or just because we haven't found a good enough scalar method? The answer is the latter.

Gumbel-Softmax: making discrete choices learnable

GSQ comes from a team at ISTA (Institute of Science and Technology Austria), ETH Zurich, and Red Hat AI. Its key innovation is using Gumbel-Softmax sampling to tackle the core difficulty of scalar quantization.

Analogy: compressing the number 0.73 to 3 bits means choosing one of 8 values (-3.5, -2.5, -1.5, -0.5, 0.5, 1.5, 2.5, 3.5). The natural choice is rounding to the nearest value — that's what GPTQ does. GSQ instead assigns each candidate a probability and samples from that distribution. Crucially, the distribution is differentiable, so it can be optimized with gradient descent — like choosing a restaurant not by distance alone, but by a learnable composite score over distance, rating, and price.

Results

On Llama-3.1-8B and 70B, GSQ is impressive:

At 2 bits: state-of-the-art among scalar quantization methods, substantially narrowing the gap to vector quantization (QTIP).
At 3 bits: the scalar-vs-vector accuracy gap is almost fully closed.
Ternary (1.58-bit) quantization: GSQ even outperforms scalar baselines running at higher bit-widths.

Inference speed on Llama-3.1-70B:

| Configuration | Average bits | Speedup | |------|---------|---------| | BF16 (original) | 16.00 | 1.00x | | Uniform 3-bit | 3.00 | 4.80x | | Non-uniform 2.62-bit | 2.62 | 4.99x | | Non-uniform 2.37-bit | 2.37 | 5.46x | | Uniform 2-bit | 2.00 | 6.20x |

A 6.2x speedup at 2 bits means a 70B model that previously required 4 GPUs might now run on one.

Scaling to trillion-parameter models

GSQ was also tested on Kimi-K2.5, a 1-trillion-parameter mixture-of-experts (MoE) model. At that scale, vector quantization is nearly inapplicable — its compute and memory overheads are too large. GSQ only requires per-coordinate discrete optimization and per-group scale parameters, so its memory footprint is close to standard scalar quantization and it scales easily to a trillion parameters. On Kimi-K2.5, 2-bit GSQ maintains strong results on math reasoning, science QA, and coding benchmarks — especially math and code, aided by a calibration dataset (OpenThoughts) dominated by math and code content.

Why it matters

GSQ's significance goes beyond the numbers: it shows the scalar quantization accuracy gap is not a fundamental limit, but an optimization problem. Moreover, GSQ uses a symmetric scalar grid, fully compatible with existing scalar inference kernels (e.g., vLLM's Humming kernels) — no custom CUDA kernels or rewritten inference engines needed. It's like finding a shortcut: instead of buying a faster car (vector quantization), just plan the route better (GSQ) to reach nearly the same speed.

Honest assessment

GSQ has limitations. It is a post-training quantization (PTQ) method requiring calibration data. Quantizing Llama-3.1-70B reportedly takes 68 hours on an H200 GPU — a nontrivial cost. At extreme long contexts (128k–256k tokens), 2-bit performance degrades somewhat. Still, GSQ is a significant advance in scalar quantization, bringing "running large models on consumer hardware" one step closer.

---

Paper: arxiv.org/abs/2604.18556