*English translation of a zhichai.net forum post analyzing the BWLA paper.*
1. Starting from a Concrete, Seemingly Absurd Question
Suppose you have a pile of numbers following a bell curve — high in the middle, low at the edges. Now map each number to one of two values: -1 or +1. The natural approach: above 0 maps to +1, below to -1. But if the curve is centered near 0, most numbers cluster around zero. +0.1 becomes +1, -0.1 becomes -1. The error is 0.9. You lose enormous information.
This is the core dilemma of LLM binarization quantization. LLM weights — hundreds of billions of parameters — follow exactly such a unimodal quasi-Gaussian distribution. Direct binarization? Disaster.
But what if you could make the distribution bimodal? Two sharp peaks, one near -1, one near +1. Then mapping to {-1, +1} becomes nearly perfect, with error approaching zero.
The question: how do you twist a unimodal distribution into a bimodal one — without retraining the model?
This is what BWLA (Binarized Weights and Low-bit Activations) answers. The work by Zhixiong Zhao, Zukang Xu, and Dawei Yang is, in my view, a small revolution in information geometry.
2. The Essence: Not Bit Count, But Distribution Shape
Quantization is not a compression problem. Compression shrinks a 100MB file to 10MB. Quantization represents real numbers with minimal bits while preserving model behavior.
Key insight: the bit count doesn't matter — what matters is the match between the distribution and the codebook.
Imagine two boxes of marbles. One box has sizes uniformly spread between 1cm and 2cm. The other has only two sizes: 0.9cm and 2.1cm. Sort both into "small" and "large" — which is easier? The second, because the marbles naturally cluster into two separated regions.
BWLA's core idea: LLM weights look like the first box (unimodal); we turn them into the second box (bimodal), and binarization becomes natural. As the paper puts it:
> "A bimodal distribution aligns naturally with the binary codebook ({−1,+1}), forming two distinct clusters that theoretically minimize quantization error."
This is my "cargo-cult detection" moment. Many chase "1-bit weights" as a magic number. But 1-bit isn't magic — matching the codebook is.
3. OKT: Twisting the Distribution with Orthogonal Transforms
BWLA's first core component is OKT (Orthogonal–Kronecker Transformation).
Imagine a crumpled sheet of paper with a bell curve drawn on it. You want to twist it into two peaks by rotating, stretching, folding — but the paper's area must stay constant: information cannot be created or destroyed.
An orthogonal transform is that rotation. Its beautiful property: R⁻¹ = Rᵀ. You can rotate (transform weights), binarize, then rotate back with Rᵀ at inference — model output unchanged. This is the paper's "preserving forward-pass equivalence."
But orthogonal matrices are large. For a 4096×4096 weight matrix, a full orthogonal matrix is also 4096×4096. OKT's cleverness: Kronecker factorization — decomposing one large orthogonal matrix into the outer product of two smaller ones. If H and R are small orthogonal matrices, H⊗R is orthogonal, with parameters dropping from n² to n₁² + n₂² (n₁×n₂ = n). The paper uses n₁/n₂ = 1, making overhead nearly negligible.
EM-Style Conditional Minimization
OKT's optimization is gradient-free, EM-style:
- E-step: given the current rotation, assign weights to the two cluster centers (-1 and +1).
- M-step: given the assignments, find the optimal rotation matrix.
- BWLA achieves up to 37% perplexity reduction on LLaMA
- On Qwen3, BiLLM and ARB-LLM "nearly collapse," while BWLA stays strong
- BWLA's effective weight precision increases by less than 0.1 bit
- FP16 activations: BWLA retains ~75% of full-precision performance, even exceeding 3-bit GPTQ at less than half the memory
- A6: existing methods completely collapse — MMLU near random guess (25%), HumanEval and GSM8K drop to 0 — while BWLA retains ~94% of its unquantized-activation performance
- QAT (quantization-aware training): simulates low-precision computation during training. Effective but expensive.
- PTQ (post-training quantization): quantizes after training, no retraining. Cheap but historically poor at extreme low bit.
Iterate to convergence. No backpropagation, no end-to-end training — which is why it works in the PTQ setting, with dozens of lightweight iterations instead of hours of gradient descent.
Experiment: LLaMA2-7B's total OKT+PSP optimization time is only 0.10 hours, versus 1.6h for OmniQuant and 0.3h for OSTQuant — 16× faster than OmniQuant, 3× faster than OSTQuant.
4. PSP: Absorbing the Stubborn Outliers with SVD
OKT twists most weights into bimodal shape, but some outliers resist. BWLA's second component, PSP (Proximal SVD Projection), handles them:
1. Compute residual error in the OKT-transformed coordinate space 2. Approximate that error with a low-rank matrix M (via truncated SVD) 3. Add M to the binarized weights as a correction term
SVD finds "base patterns" ranked by importance; truncated SVD keeps only the top few. The "proximal" part uses an upper bound guaranteeing monotone objective descent each iteration — no learning rate tuning needed.
PSP's rank ratio is 0.005: for a 4096×4096 matrix, rank ~20, adding under 0.5% extra parameters. With PSP, residual outliers are significantly suppressed and bimodal symmetry greatly improved.
5. Why One Transformation Handles Both Weights and Activations
This is the most elegant detail. OKT transforms weights: W̃ = Rᵀ·W; after binarization, invert: W_deq = binarize(W̃)·R. Since R⁻¹ = Rᵀ, forward-pass equivalence holds.
What about activations? LLM activation outliers are another major quantization problem. BWLA's answer: the same Rᵀ applies to activations.
Why? In a Transformer, activations X and weights W interact via Y = X·W. If W is orthogonally transformed, activations must be transformed identically to keep the computation consistent. As the paper says: "the same transformation can be applied to activations while preserving forward-pass equivalence."
Not two independent tricks, but one unified orthogonal framework solving both weight distribution shape and activation outliers.
6. Experimental Results: When Competitors Collapse, BWLA Stands
6.1 Weight-only binarization (A16)
With 16-bit activations, the paper reports BWLA "improves average accuracy by 13% and reduces perplexity by 28% compared with current state-of-the-art binary methods" (BiLLM, ARB-LLM, DBellQuant). Under the harder A6 (6-bit activation) setting, the gap widens dramatically:
6.2 Reasoning in instruct models
Qwen3-32B-Instruct on MMLU, HumanEval, GSM8K:
When everyone else falls, BWLA keeps running.
7. Where BWLA Sits: PTQ vs QAT
8. Limitations and My Unanswered Questions
Honest caveats:
1. Generalization of the orthogonal transform — OKT learns from calibration data; sensitivity to calibration set choice isn't discussed. 2. Extremely low-bit activations (A4 or lower) — only A16 and A6 are shown; robustness below that is untested. 3. MoE models — the dominant architecture trend; not addressed. 4. Real inference speed — the paper focuses on accuracy; kernel optimization, memory bandwidth, and CPU-GPU issues are not discussed.
These aren't criticisms — they're directions for follow-up work.
9. A Feynman-Style Takeaway: The Aesthetics of Information Geometry
Quantization isn't compression. BWLA doesn't try to "approximate" original weights — it changes the coordinate system so information naturally matches the binary codebook.
It's like translation: translating a book into a two-word language directly loses everything. But if you first find an "intermediate language" where the content is effectively expressible with two words, the translation is nearly lossless. OKT is the search for that intermediate language; orthogonal transforms are the translation; Kronecker factorization makes the dictionary pocket-sized.
The paper is titled "Breaking the Barrier" — aptly. BWLA breaks not a numeric record but our intuition about what PTQ can do.
Closing Thought
What excites me isn't the numbers — it's the framing: when everyone tried to approximate a unimodal distribution with ever-more-complex algorithms, BWLA asked a more fundamental question: what if the distribution itself isn't the problem — what if we chose the wrong coordinates?
Change the coordinates, and unimodal becomes bimodal. Binarization goes from impossible to natural. This isn't a victory of algorithms — it's a victory of perspective.
"The first principle is that you must not fool yourself." Many were fooled by the name "1-bit weights" into thinking bits are the goal. BWLA reminds us: the goal is matching — distribution to codebook, information to representation.
---
References
1. BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs — Zhixiong Zhao, Zukang Xu, Dawei Yang. arXiv:2605.00422, 2026-04-29. https://arxiv.org/abs/2605.00422
2. BiLLM — Huang et al. arXiv:2402.04291, 2024.
3. ARB-LLM — Li et al. arXiv:2410.03129, 2024.
4. DBellQuant — Ye et al. arXiv:2507.01027, 2025.
5. OSTQuant — Hu et al. arXiv:2501.13987, 2025.
6. BitNet b1.58 — Wang et al. arXiv:2402.17764, 2024.
7. GPTQ — Frantar et al. arXiv:2210.17323, 2022.
8. AWQ — Lin et al. arXiv:2306.00978, 2023.
9. QuIP# — Tseng et al. arXiv:2402.04396, 2024.
10. Qwen3 Technical Report — Yang et al. arXiv:2505.09388, 2025.