Overview
This forum post analyzes BWLA (Binarized Weights and Low-bit Activations), a paper (arXiv:2605.00422) by Zhixiong Zhao, Zukang Xu, and Dawei Yang proposing the first pure post-training quantization (PTQ) framework for W1AX — 1-bit weights with low-bit activations — without retraining.
Key points
- Core insight: Quantization error depends on how well the weight distribution matches the codebook, not on bit count alone. LLM weights are unimodal and quasi-Gaussian, so naive binarization to {-1, +1} destroys information for values clustered near zero. If the distribution were bimodal (peaks near -1 and +1), binarization would be nearly lossless.
- OKT (Orthogonal-Kronecker Transformation): An orthogonal rotation R reshapes weights (W̃ = Rᵀ·W). Because R⁻¹ = Rᵀ, the forward pass remains exactly equivalent after dequantization. Kronecker factorization (H⊗R) reduces parameters from n² to n₁² + n₂², making the transform nearly free in memory and compute.
- Gradient-free optimization: OKT uses EM-style conditional minimization — assign weights to cluster centers (E-step), find the optimal rotation (M-step) — enabling fast PTQ. Total OKT+PSP optimization on LLaMA2-7B takes only 0.10 hours, vs 1.6h for OmniQuant and 0.3h for OSTQuant.
- PSP (Proximal SVD Projection): Residual outliers are absorbed by a low-rank matrix obtained via truncated SVD (rank ratio 0.005, <0.5% extra parameters), acting as a correction term on the binarized weights. The proximal formulation guarantees monotonic objective decrease without learning-rate tuning.
- Unified weight/activation treatment: The same orthogonal transform applies to activations, since Y = X·W — transforming both preserves equivalence and mitigates activation outliers simultaneously.
- Compared with state-of-the-art binary PTQ methods (BiLLM, ARB-LLM, DBellQuant), BWLA reportedly improves average accuracy by 13% and reduces perplexity by 28%.
- In the harder A6 (6-bit activation) setting: up to 37% perplexity reduction on LLaMA; BiLLM and ARB-LLM "nearly collapse" on Qwen3 while BWLA stays strong, with effective weight precision increased by less than 0.1 bit.
- On Qwen3-32B-Instruct (MMLU, HumanEval, GSM8K): BWLA retains ~75% of FP16 performance with A16 — even exceeding 3-bit GPTQ at less than half the memory — and ~94% relative performance with A6, where prior methods drop to near-random or zero accuracy.
- Unlike BitNet b1.58, which requires quantization-aware training (QAT), BWLA shows a binarization-friendly structure can be exposed purely through orthogonal re-coordination of the weight space — a PTQ result.
- The author notes open questions: sensitivity to calibration data, robustness at A4/A2 activations, applicability to MoE models, and real-world inference kernels/speed (the paper focuses on accuracy).
- BWLA: https://arxiv.org/abs/2605.00422
- BiLLM: https://arxiv.org/abs/2402.04291
- ARB-LLM: https://arxiv.org/abs/2410.03129
- DBellQuant: https://arxiv.org/abs/2507.01027
- OSTQuant: https://arxiv.org/abs/2501.13987
- BitNet b1.58: https://arxiv.org/abs/2402.17764
- GPTQ: https://arxiv.org/abs/2210.17323
- AWQ: https://arxiv.org/abs/2306.00978
- QuIP#: https://arxiv.org/abs/2402.04396
- Qwen3 Technical Report: https://arxiv.org/abs/2505.09388