EdgeRazor Deep Dive: How 1.58-bit Mixed-Precision Quantization Delivers 15x Faster On-Device LLM Inference

Forum topic · 小凯 · 2026-06-11

Summary

EdgeRazor is a lightweight framework from Nanjing University and Microsoft AI for compressing large language models via mixed-precision quantization-aware distillation. It combines three modules: structural mixed-precision quantization (SQMP) that allocates 4-bit precision to sensitive channels and ternary 1.58-bit weights elsewhere using a periodic super-group scheme; layer-adaptive feature distillation (LAFD) that automatically selects the most informative teacher layers for supervision; and entropy-aware KL divergence (EAKLD) that dynamically balances forward and reverse KL based on teacher confidence. On Qwen3-0.6B, 1.58-bit EdgeRazor shrinks storage from 1.11 GB to 0.19 GB and achieves a 15.16x decoding speedup on Apple M4 Pro via llama.cpp. Its 1.88-bit configuration outperforms all 3-bit baselines by 4.38 points and SOTA 2-bit PTQ by 11.27 points, while training costs 4-10x fewer tokens than ParetoQ. The framework also extends to multimodal models like Qwen2.5-Omni-7B, matching BF16 accuracy at 4-bit.

EdgeRazor: 1.58-bit Quantization for Practical On-Device LLMs

Paper: EDGERAZOR: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation arXiv: https://arxiv.org/abs/2605.04062 | Code: https://github.com/zhangsq-nju/EdgeRazor Authors: Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng, Xin-Yi Zou, Chen Wu, Nan Li, Shao-Qun Zhang, Zhi-Hua Zhou (Nanjing University State Key Lab for Novel Software Technology / School of AI, Microsoft AI)

Key points

Goal: Make ultra-low-bit quantization (1.58-bit ternary weights) actually usable for on-device LLM inference, where storage and compute are the main bottlenecks (Qwen3-0.6B FP16 weights: 1.11 GB).
SQMP (mixed precision): An adjustable ratio ρ ∈ [0,1] of weights stay 4-bit while the rest are ternary 1.58-bit. Channels are arranged in periodic super-groups (1 of every ⌊1/ρ⌋ consecutive output channels at 4-bit), which is proven optimal (error Θ(N⁻¹) vs Θ(1) for stacked allocation) and aligns with hardware granularity.
LAFD (layer-adaptive feature distillation): Automatically picks the k=3 teacher layers with lowest inter-layer cosine similarity (i.e., most informative) for MSE feature distillation, beating fixed-layer selection by up to 1.52 points.
EAKLD (entropy-aware KL): Weights forward vs reverse KL dynamically via λ = mean(min(H(P_T), log K)/log K): high teacher entropy → mode-covering, low entropy → mode-seeking. Outperforms BitDistiller's CAKLD by up to 1.12 points.

Benchmark highlights (Qwen3-0.6B)

| Method | Precision | Avg score | vs BF16 (47.35) | |---|---|---|---| | AQLM | 2-bit W-only | 36.51 | -10.84 | | EdgeRazor | 1.88-bit W-only | 41.60 | -5.75 | | FlatQuant | 3-bit W8A8 | 37.38 | -9.97 | | EdgeRazor | 1.88-bit W8A8 | 41.76 | -5.59 |

1.88-bit EdgeRazor beats all 3-bit baselines by 4.38 points and SOTA 2-bit PTQ by 11.27 points.
Vs ParetoQ (QAT SOTA) on MobileLLM-350M: comparable or better accuracy with 4-10x lower training budget (e.g., 1.88-bit: 3.1B vs 30B tokens).

Deployment (llama.cpp, Apple M4 Pro)

| Metric | 1.58-bit EdgeRazor | BF16 | Gain | |---|---|---|---| | Storage | 0.19 GB | 1.11 GB | 5.8x | | Memory | 0.51 GB | 1.46 GB | 2.9x | | Prefilling | 711.67 tok/s | 337.99 tok/s | 2.11x | | Decoding | 317.03 tok/s | 20.91 tok/s | 15.16x |

Speedup comes from ternary weights enabling lookup tables instead of multiplications, SIMD-friendly periodic structure, and llama.cpp's native TQ2_0 format.

Multimodal (Qwen2.5-Omni-7B, 4-bit)

Matches BF16 on Video-MME (62.22) and MLVU (48.82), beating AWQ and GPTQ.

Limitations

Requires an FP16 teacher model for distillation.
Hyperparameters (ρ, loss weights, k) need per-model tuning; per-layer ρ was less stable than a global value.
Verified only up to 7B models; long-context KV cache effects untested.
1.58-bit support is currently limited to llama.cpp-style runtimes; power/thermal constraints on phones remain.

Takeaway

EdgeRazor shows 1.58-bit quantization is practical, not a toy: a 0.6B model at 300+ tokens/s decoding on an iPhone lowers the barrier for offline writing assistants, local knowledge bases, and privacy-sensitive on-device AI.