BWLA: Twisting LLM Weights into a Bimodal Distribution — an Information-Geometry Take on W1AX Post-Training Quantization

1. Starting from a seemingly absurd question

Suppose you have a pile of numbers distributed like a bell curve — high in the middle, tapering at the edges. Now map every number to one of two values: -1 or +1. The natural approach is sign-based mapping: positives become +1, negatives become -1. But if the bell curve is centered near zero, most numbers cluster around 0. Mapping +0.1 → +1 and -0.1 → -1 creates an error of 0.9 per value — a massive information loss.

This is the core dilemma of binary quantization for LLMs. LLM weights — the hundreds of billions of parameters that determine model behavior — are precisely unimodal, quasi-Gaussian distributions. Direct binarization is a disaster.

But what if the distribution could be reshaped into bimodal — two sharp peaks near -1 and +1? Then mapping to {-1, +1} becomes nearly lossless. The question: how do you twist a unimodal distribution into a bimodal one without retraining the model?

This is what BWLA (Binarized Weights and Low-bit Activations) answers. The work by Zhixiong Zhao, Zukang Xu, and Dawei Yang is, in my view, a small revolution in information geometry.

2. The real problem: not bit count, but distribution shape

Quantization is not compression. Compression shrinks a 100 MB file to 10 MB. Quantization represents real numbers with the fewest bits while preserving model behavior.

The key insight: the number of bits matters less than the match between distribution and codebook.

Imagine two boxes of marbles. One box has sizes uniformly spread between 1 cm and 2 cm. The other has only two sizes: 0.9 cm and 2.1 cm. Which box is easier to classify with just "small" and "large" labels? The second — the marbles naturally cluster into two separated groups.

BWLA's core idea is exactly this: LLM weights are like the first box (unimodal); reshape them into the second box (bimodal), and binarization becomes natural. As the paper puts it:

> "A bimodal distribution aligns naturally with the binary codebook ({−1,+1}), forming two distinct clusters that theoretically minimize quantization error."

This is my "cargo-cult detection" moment. Many people chase "1-bit weights" as a magic number. But 1 bit isn't magic — matching the codebook is. If you don't understand this, you're the person building a bamboo control tower: formally correct, but no planes will come.

3. OKT: twisting the distribution with orthogonal transforms

BWLA's first core component is OKT (Orthogonal–Kronecker Transformation).

Picture a crumpled sheet of paper with a bell curve drawn on it. You want to twist it into two peaks. You can rotate, stretch, and fold it — but the paper's area must stay constant: information can't be created or destroyed.

An orthogonal transform is exactly this "rotation." Its beautiful mathematical property: R⁻¹ = Rᵀ. The inverse of a rotation is its transpose. So you can rotate (transform the weights), binarize, then rotate back at inference time (using Rᵀ), and the model's output is unchanged. This is the paper's "preserving forward-pass equivalence."

But orthogonal matrices are large. For a 4096×4096 weight matrix, a full orthogonal matrix is also 4096×4096. OKT's cleverness lies in Kronecker factorization: decomposing one large orthogonal matrix into the Kronecker product of two smaller ones. If H and R are small orthogonal matrices, H ⊗ R is still orthogonal, but parameters drop from n² to n₁² + n₂² where n₁ × n₂ = n. The paper uses n₁/n₂ = 1 (equal-sized factors), making overhead nearly negligible.

EM-style conditional minimization

OKT is also gradient-free, using EM-style conditional minimization:

• E-step: given the current rotation, assign weights to the two cluster centers (-1 and +1).
• M-step: given the assignments, find the optimal rotation matrix.

Iterate until convergence. No backpropagation, no end-to-end training — which is why it works in the PTQ setting with only lightweight iterations instead of hours of gradient descent.

Benchmark: OKT+PSP total optimization time on LLaMA2-7B is just 0.10 hours, versus 1.6 hours for OmniQuant and 0.3 hours for OSTQuant — 16× faster than OmniQuant and 3× faster than OSTQuant.

4. PSP: absorbing the stubborn outliers with SVD

OKT twists most weights into a bimodal shape, but some "stubborn elements" — outliers — remain far from the bimodal centers no matter how you rotate.

BWLA's second component, PSP (Proximal SVD Projection), handles them:

1. In the post-OKT coordinate space, compute the residual error. 2. Approximate this error with a low-rank matrix M (via truncated SVD). 3. Add M to the binarized weights as a correction term.

SVD finds the most important "basis patterns" of a matrix, ranked by significance; truncated SVD keeps only the top few, discarding noise. The "proximal" part means PSP uses an upper-bound surrogate that guarantees monotone objective descent per iteration — no hand-tuned learning rate needed.

PSP's rank ratio is 0.005 — for a 4096×4096 matrix, rank ≈ 20, under 0.5% extra parameters. Yet experimentally, the residual outliers are significantly suppressed and the bimodal symmetry improves substantially.

5. Why one transform handles both weights and activations

The detail I find most elegant in BWLA.

OKT transforms weights: W̃ = Rᵀ · W, binarizes, then maps back: W_deq = binarize(W̃) · R. Since R⁻¹ = Rᵀ, the forward pass is fully equivalent.

But what about activations? LLM activations also suffer from extreme outliers that cause catastrophic low-bit quantization error. BWLA's answer: the same Rᵀ can be applied to activations.

Why? In the Transformer forward pass, activations X interact with weights W via matrix multiplication: Y = X · W. If W is orthogonally transformed, activations must undergo the same transform to keep the computation consistent. As the paper states: "the same transformation can be applied to activations while preserving forward-pass equivalence."

The symmetry is beautiful: not two independent tricks, but one unified orthogonal framework solving both the weight-distribution problem and the activation-outlier problem.

6. Experimental results: when competitors collapse, BWLA stands

Key numbers from the paper. Note: for perplexity (PPL), lower is better.

6.1 Weight-only binarization (A16)

BWLA improves average accuracy by 13% and reduces perplexity by 28% compared with current state-of-the-art binary PTQ methods (BiLLM, ARB-LLM, DBellQuant).

Under the harder A6 (6-bit activation) setting, the gap widens dramatically:

• BWLA achieves perplexity reductions of up to 37% on LLaMA.
• On Qwen3, BiLLM and ARB-LLM nearly collapse, while BWLA stays strong.
• BWLA's effective weight precision increases by less than 0.1 bit.

6.2 Instruction-model reasoning capability

Qwen3-32B-Instruct evaluated on MMLU, HumanEval, and GSM8K:

FP16 activations:

• BWLA retains roughly 75% of full-precision performance.
• It even outperforms 3-bit GPTQ while using less than half the memory.

A6 (6-bit activations):

• Existing methods collapse almost completely: MMLU accuracy near random guessing (25%); HumanEval and GSM8K drop to 0.
• BWLA retains roughly 94% of its performance relative to unquantized activations.

That's what I mean by "standing": when everyone else falls over, BWLA keeps running.

7. Positioning: the PTQ vs QAT boundary

• QAT (Quantization-Aware Training): simulates low-precision computation during training so the model adapts to quantization error. Effective, but requires expensive end-to-end training.
• PTQ (Post-Training Quantization): quantizes after training with no retraining. Cheap, but typically much worse — especially at extreme bits (1-bit weights, low-bit activations).

BitNet b1.58 needs QAT for similar effects. BWLA proves that without retraining, merely reorganizing the geometry of weights and activations can approach QAT-level results. This is a significant theoretical finding: it suggests LLM weight spaces contain an intrinsic, binarization-friendly structure that orthogonal transforms can expose — a structure that isn't learned, but was always there, hidden by the unimodal appearance of the distribution.

8. Limitations and my unanswered questions

Honestly, BWLA isn't perfect. Several questions go unanswered or underexplored:

1. Generalization of the orthogonal transform: OKT's rotation is learned from calibration data. If calibration data doesn't represent real usage distributions, transform quality degrades. The paper doesn't discuss calibration-data sensitivity or selection strategy in depth. 2. Extremely low-bit activations (A4 or lower): results cover A16 and A6. Does OKT+PSP remain robust at A4 or A2? No answer given. 3. Applicability to MoE models: MoE is now mainstream. How does the framework handle sparsely activated expert models? Not covered. 4. Actual inference speed: the paper focuses on accuracy. Kernel optimization, memory-bandwidth bottlenecks, and CPU-GPU coordination for 1-bit weights are undiscussed.

These aren't criticisms — they're directions for future work.

9. A Feynman-style view: the aesthetics of information geometry

Quantization is not compression. BWLA's aesthetic core: it doesn't try to "approximate" the original weights — it changes the coordinate system so the original information naturally matches the binary codebook.

It's like translation. Translating a book into a minimalist two-word language loses enormous information — unless you first find an intermediate language in which the book's content is naturally expressible with two words. OKT is the search for that intermediate language. Orthogonal transforms are the translation; Kronecker factorization makes the dictionary small enough to fit in your pocket.

That's what I call information geometry. It's not about bit count — it's about the shape of information.

The paper is titled "Breaking the Barrier," and the title fits. BWLA breaks not a numerical record, but our intuition about what PTQ can do.

Closing thoughts

What excites me about BWLA isn't the numbers — impressive as they are. It's the thinking: while everyone tries to build more sophisticated quantization algorithms to "approximate" a unimodal distribution, BWLA asks a more fundamental question: what if the distribution itself isn't the problem? What if the problem is that we chose the wrong coordinate system?

Change coordinates, and unimodal becomes bimodal. Binarization goes from impossible to natural. This isn't a victory of algorithms — it's a victory of perspective.

"The first principle is that you must not fool yourself." Many people are fooled by the name "1-bit weights" into thinking the goal is bit count. BWLA reminds us: the goal is matching. Distribution matching codebook. Information matching representation.

That's what it means to truly understand a concept, not just remember its name.

---

References

1. BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs — Zhixiong Zhao, Zukang Xu, Dawei Yang. arXiv:2605.00422, 2026-04-29. https://arxiv.org/abs/2605.00422 2. BiLLM: Pushing the Limit of Post-Training Quantization for LLMs — Huang et al. arXiv:2402.04291, 2024. 3. ARB-LLM: Alternating Refined Binarizations for Large Language Models — Li et al. arXiv:2410.03129, 2024. 4. DBellQuant: Breaking the Bell with Double-Bell Transformation — Ye et al. arXiv:2507.01027, 2025. 5. OSTQuant: Refining LLM Quantization with Orthogonal and Scaling Transformations — Hu et al. arXiv:2501.13987, 2025. 6. BitNet b1.58: 1-bit LLMs — Wang et al. arXiv:2402.17764, 2024. 7. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Frantar et al. arXiv:2210.17323, 2022. 8. AWQ: Activation-aware Weight Quantization — Lin et al. arXiv:2306.00978, 2023. 9. QuIP#: Even Better LLM Quantization with Hadamard Incoherence — Tseng et al. arXiv:2402.04396, 2024. 10. Qwen3 Technical Report — Yang et al. arXiv:2505.09388, 2025.