Low-Bit N-Gram Embeddings for Language Models: 4-Bit Tables Cut 75% Memory with Minimal Loss

Forum topic · 小凯 · 2026-10-24

Summary

A new arXiv paper (2610.31646) by Stephen Chung and Wenyu Du explores compressing n-gram embedding tables used in modern open-weight language models. These tables store learned representations of short token sequences and can consume enormous memory — DeepSeek-V4.1-Flash, for instance, dedicates 196 billion parameters to its Engram n-gram memory. The authors propose storing these embeddings at low precision, systematically evaluating quantization methods and bit allocation strategies across tables. Experiments training nanochat, Llama, and GPT-2 family models from scratch show that 4-bit tables save roughly 75% of table memory while retaining over 99% of the validation-loss improvement that the tables provide. Lower bit-widths degrade performance more, but short fine-tuning recovers a substantial portion. Reinvesting saved memory into larger tables can even outperform the original full-precision setup. On existing large models (DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next), low-bit tables save 47-55% of table memory with perplexity increases below 1.2% and downstream task performance largely intact.

Paper Overview

Field: NLP
Authors: Stephen Chung, Wenyu Du
Published: 2026-10-22
arXiv: 2610.31646

Abstract (translated)

Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences (such as consecutive two or three tokens), but they can require substantial memory: for example, DeepSeek-V4.1-Flash allocates 196 billion parameters to its Engram n-gram memory. The authors therefore investigate whether this memory cost can be reduced while preserving the performance improvement.

They propose storing n-gram embeddings at low precision and systematically study different quantization methods and bit allocation strategies across tables. Results from training nanochat, Llama, and GPT-2 family models from scratch show that 4-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables.

Using fewer bits leads to larger performance degradation, but short fine-tuning recovers a substantial part of it. Moreover, reinvesting the saved memory into larger tables can surpass the performance of the original full-precision tables while still using less memory.

These savings also apply to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases below 1.2% and downstream task performance largely maintained.

Key Takeaways

4-bit n-gram tables: ~75% table memory savings, >99% of validation-loss benefit retained.
Aggressive quantization is recoverable: short fine-tuning restores much of the loss from lower bit-widths.
Memory reinvestment: larger low-bit tables can beat the original full-precision baseline at lower total memory.
Works on deployed-scale models: 47-55% memory savings on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next with <1.2% perplexity increase.

---

*Auto-collected on 2026-10-24.*