A 'Memory Book' Worth 196 Billion Parameters
Imagine memorizing vocabulary. Every time you learn a new word, you also note how it combines with the previous one or two words — 'coffee' is likely followed by 'shop' or 'cup', while what follows 'hit' feels completely different whether it's 'phone call' or 'basketball'.
This 'chain-the-words-together' intuition is exactly what an n-gram embedding table in a language model stores: representations for combinations of consecutive tokens. DeepSeek-V4.1-Flash allocates 196 billion parameters to this table — more than many complete models have.
Here's the question: do all 196 billion parameters really need to be stored in 16-bit floating point?
Stephen Chung and Wenyu Du from DualverseAI answer: no — 4 bits are enough.
Core Finding: 4 Bits Save 75% Memory, Retain 99% Performance
The paper's systematic experiments span three model families (nanochat, Llama, GPT-2), all trained from scratch, with clean conclusions:
| Bits | Memory Saved | Performance Retained | |--------|---------|---------| | 4-bit | ~75% | >99% | | 3-bit | ~80% | ~97% | | 2-bit | ~87% | Notable drop, but recoverable via fine-tuning |
Key number: the 4-bit table loses less than 1% of the validation-loss improvement while saving three quarters of the memory.
On DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables saved 47-55% of table memory with less than 1.2% perplexity increase, while downstream task performance was largely preserved.
How It Works: Codebooks + Scale Factors
The core idea of low-bit quantization is simple: instead of storing full 16-bit float vectors, store an index pointing into a small shared array (a codebook).
The paper compares three codebook constructions:
- Uniform codebook: equal-spacing partitioning
- Normal-quantile codebook: partitions based on quantiles of a normal distribution
- Lloyd-Max codebook: the classic optimal scalar quantization method
- Much lower deployment barriers: the same GPU can run larger models
- Lower inference cost: memory bandwidth is a major inference bottleneck
- Bigger n-gram tables: the same memory budget fits a table 4x larger
- Experiments are mainly on small-to-medium models (up to 3B). While validation was done on larger models, the 'reinvestment' experiment wasn't run at DeepSeek scale
- Codebook choice and mixed-precision strategy need per-model tuning; there's no 'one-click optimum'
- Compression below 2 bits requires fine-tuning; it's not entirely training-free
Each embedding vector also carries a 16-bit scale factor to restore the original vector's magnitude. This lets even 2-4 bit indices express sufficiently rich values.
The paper also explores mixed precision: different tables get different bit widths. Important tables get more bits, less important ones fewer.
The Most Interesting Finding: Saved Memory Can Be 'Reinvested'
The paper's most compelling section isn't compression itself but a 'reinvestment' experiment:
> Using the memory saved by low-bit quantization to build larger tables not only matched the original full-precision tables but exceeded them — while total memory was still lower.
It's like urban planning: redeveloping underutilized land at higher density ends up housing more people than before. Low-bit quantization isn't just 'saving money' — it's 'reallocating resources to more effective places'.
Another practical finding: 2-bit tables degrade badly, but brief fine-tuning on data recovers most of the performance. If extreme compression is a hard requirement (e.g., edge deployment), 2-bit + fine-tuning is a viable path.
Why This Matters
N-gram embedding tables are becoming standard in large models. The latest DeepSeek and Qwen versions all use them, because they improve model quality at minimal compute cost — table lookup is O(1). The cost is memory bloat: a 196-billion-parameter table in 16-bit precision takes 392 GB.
If 4 bits retain 99% of the benefit, that means:
Honest Assessment
The paper's limitations are clear:
The answer: fewer than most people expect.
---
Paper: arxiv.org/abs/2610.31646 Code: github.com/dualverse-ai/lowbit-ngram