EDGERAZOR Deep Dive: How 1.58-bit Mixed-Precision Quantization Delivers 15x Faster On-Device LLM Inference

Key points

EDGERAZOR (arXiv: 2605.04062) is a quantization-aware distillation framework from Nanjing University and Microsoft AI that makes extreme low-bit (1.58-bit) quantization of LLMs practical for on-device deployment.

Problem

• On-device LLMs are bottlenecked by storage and compute: Qwen3-0.6B FP16 needs 1.11 GB storage / 1.46 GB memory.
• Below 4-bit, post-training quantization (PTQ) collapses; quantization-aware training (QAT) works but is expensive.

Three core modules

• SQMP (Structural Mixed-Precision Quantization): a tunable parameter rho distributes a fraction of output channels at 4-bit, the rest at 1.58-bit (ternary {-1, 0, +1}). Channels are assigned in periodic super-groups (1 of every floor(1/rho) consecutive channels at 4-bit), which the paper proves gives error Theta(N^-1) vs Theta(1) for stacked allocation and O(N^-1/2) for random allocation, while aligning with hardware granularity.
• LAFD (Layer-Adaptive Feature Distillation): automatically picks the k=3 teacher layers with lowest inter-layer cosine similarity (most informative) for MSE feature distillation, instead of hand-picked fixed layers. Ablations show +0.44 to +1.52 points over fixed-layer selection.
• EAKLD (Entropy-Aware KL Divergence): dynamically blends forward and reverse KLD using teacher entropy (lambda = mean(min(H(P_T), log K)/log K)): high entropy -> mode-covering, low entropy -> mode-seeking. Outperforms BitDistiller's CAKLD by up to +1.12 points.

Key results

• Qwen3-0.6B: 1.88-bit EdgeRazor scores 41.60 (weight-only), beating all 2-bit PTQ methods (AQLM 36.51, QTIP 35.94) by 11.27 points; 1.88-bit weight-activation quantization (41.76) beats all 3-bit baselines (FlatQuant 37.38, OmniQuant 34.58) by 4.38 points.
• Training budget: matches or beats ParetoQ (QAT SOTA) on MobileLLM-350M with 4-10x fewer training tokens (e.g., 3.1B vs 30B tokens at 1.58/1.88-bit).
• Deployment (llama.cpp, Apple M4 Pro): 1.58-bit Qwen3-0.6B compresses 1.11 GB -> 0.19 GB (5.8x), memory 1.46 GB -> 0.51 GB, prefilling 711.67 vs 337.99 tokens/s (2.11x), decoding 317.03 vs 20.91 tokens/s (15.16x). Ternary weights enable lookup tables instead of multiplications and SIMD-friendly access patterns (llama.cpp TQ2_0 format).
• Compression ratio: 7.03x at 1.58-bit vs 2.94x for baselines, because EdgeRazor quantizes ~99.99% of weights including embeddings and lm_head (baselines keep these at FP16).
• Multimodal: 4-bit Qwen2.5-Omni-7B matches BF16 on Video-MME (62.22) and MLVU (48.82), beating AWQ and GPTQ.

Discussion

• 1.58-bit works because ternary weights give zero-value sparsity, replace multiplications with sign-flips/lookups, and mixed 4-bit channels compensate for precision-sensitive weights. The ternary scale factor uses beta * mean(|w|) (beta = 2.0) for robustness.
• Unlike BitNet (trained from scratch), EDGERAZOR compresses existing pretrained models, which is more practical and cheaper.
• Ablations confirm super-group allocation beats stacked allocation (+0.91/+1.02), and the three modules show synergistic gains.

Limitations and open directions

• Requires an FP16 teacher model; several hyperparameters (rho, loss weights, k) need per-model tuning.
• Only validated up to 7B models; long-context / KV-cache behavior untested.
• Broader framework support (vLLM, TensorRT-LLM) for 1.58-bit formats and mobile power/thermal constraints remain engineering challenges.
• Future work: dynamic per-layer rho, joint weight-activation-KV quantization, and hardware co-design for ternary arithmetic.

Conclusion

EDGERAZOR demonstrates that 1.58-bit quantization is no longer a toy: a 0.6B model at ~0.19 GB decoding at 300+ tokens/s on an iPhone-class device makes offline assistants, local knowledge bases, and privacy-sensitive on-device AI substantially more accessible.

References

• Paper: https://arxiv.org/abs/2605.04062
• Code: https://github.com/zhangsq-nju/EdgeRazor
• Authors: Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng, Xin-Yi Zou, Chen Wu, Nan Li, Shao-Qun Zhang, Zhi-Hua Zhou (Nanjing University; Microsoft AI)