Fairy2i: Ultra-Low-Bit LLM Quantization Using Complex Weights {±1, ±i}

Forum topic · ✨步子哥 · 2026-05-07

Summary

Fairy2i is a 2-bit quantization method from Peking University that converts real-valued LLaMA checkpoints into complex-valued models whose weights take only four states: {+1, -1, +i, -i}. A widely-linear transformation (y = Ux + W·conj(x)) provides a mathematically lossless bridge from real matrices to complex form, so pre-trained checkpoints can be reused instead of training from scratch. Phase-aware quantization projects each weight onto the nearest fourth root of unity, fully utilizing the 2-bit encoding space, while recursive residual quantization (T=2 stages) refines accuracy. On LLaMA-2 7B, Fairy2i-W2 achieves a C4 perplexity of 7.85 versus 6.63 for FP16, and a 62.00 zero-shot average versus 64.72 full precision, clearly outperforming GPTQ, AQLM, and binary baselines. Storage drops from ~14GB to ~3.5GB, and multiplication-free inference uses only additions, subtractions, and real/imaginary swaps, enabling efficient on-device deployment.

This post is a detailed reading note on Fairy2i (arXiv:2512.02901), a Peking University paper proposing 2-bit complex-valued LLM quantization with all parameters in {±1, ±i}.

Why complex numbers fit 2 bits perfectly

| Method | Bits | States | Problem | |---|---|---|---| | Binary {+1, -1} | 1-bit | 2 | Severe information loss | | Ternary {+1, 0, -1} | 1.58-bit | 3 | 25% of 2-bit space wasted | | Complex {±1, ±i} | 2-bit | 4 | Fully utilized, no waste |

Binary/ternary real-valued methods waste part of the 2-bit encoding space, while the four complex states map exactly onto the four quadrants of the complex plane. The core insight: encode smarter, not compress harder.

The threefold dilemma

1. Extremely low bits are needed for edge devices. 2. Existing complex models require training from scratch — they cannot reuse LLaMA/Qwen checkpoints. 3. Traditional PTQ methods (GPTQ, AQLM, QuIP#) lose significant accuracy or need lots of calibration data.

Step 1: Widely-linear transformation (lossless real → complex)

Theorem 1: any real linear transform $\tilde{y} = R\tilde{x}$ ( $R \in \mathbb{R}^{(2n) \times (2m)}$ ) is exactly equivalent to the complex widely-linear form:

y = Ux + W\bar{x}

with block-averaged parameters:

\begin{cases} \Re U = \frac{1}{2}(R_{11}+R_{22}), & \Im U = \frac{1}{2}(R_{21}-R_{12}) \\ \Re W = \frac{1}{2}(R_{11}-R_{22}), & \Im W = \frac{1}{2}(R_{12}+R_{21}) \end{cases}

This is exact equivalence, not approximation. Each real parameter needs only 1 bit afterward. On LLaMA-2 7B, Fairy2i-W2 needs ~3.5GB vs ~14GB for FP16 (75% savings); peak memory drops from 28GB to 7GB.

Step 2: Phase-aware quantization

Codebook $S_{\text{2-bit}} = \{\pm 1, \pm i\}$ , with quantization:

b(w) = \arg\max_{s \in S_{\text{2-bit}}} \Re(w\bar{s})

i.e., project each weight's phase onto the nearest fourth root of unity. Axis-wise scaling is applied separately to real and imaginary parts, and dequantization is:

\hat{w} = s_{\text{re}}b_{\text{re}} + i \cdot s_{\text{im}}b_{\text{im}}

Step 3: Recursive residual quantization

W_q \approx \sum_{t=0}^{T-1} \tilde{W}^{(t)}, \quad R^{(t+1)} = R^{(t)} - \tilde{W}^{(t)}

| Config | Bits | C4 PPL | Avg. | Gain | |---|---|---|---|---| | Fairy2i-W1 (T=1) | 1 | 11.03 | 48.66 | baseline | | Fairy2i-W2 (T=2) | 2 | 8.74 | 57.92 | +19% Avg | | Fairy2i-W3 (T=3) | 3 | 8.38 | 58.43 | +1% Avg |

T=2 is the sweet spot; gains diminish at T=3.

Results on LLaMA-2 7B

C4 perplexity: Fairy2i-W2 achieves 7.85 vs 6.63 (FP16), beating GPTQ (10.61), AQLM (8.54), and Real-Binary (11.75).

| Method | Bits | ARC-e | ARC-c | HellaSwag | PIQA | Wino | Avg. | |---|---|---|---|---|---|---|---| | LLaMA-2 FP16 | 16 | 75.59 | 43.17 | 57.06 | 77.91 | 69.85 | 64.72 | | Fairy2i-W2 | 2 | 72.73 | 39.76 | 53.33 | 76.17 | 68.03 | 62.00 | | Real-Binary | 1 | 53.32 | 22.70 | 35.57 | 66.81 | 52.64 | 46.21 |

Learning-rate scheduling matters: double decay (LR3) yields the best PPL (7.85) and average accuracy (62.00). Training uses 30B RedPajama tokens with AdamW and a WSD scheduler.

Comparison with related work

vs BitNet 1.58-bit: ternary wastes 25% of the 2-bit space; Fairy2i's complex binary values use it fully.
vs iFairy: the widely-linear transform directly reuses LLaMA checkpoints, cutting training cost to 30B tokens.
vs GPTQ/AQLM/QuIP#: Fairy2i requires continued training but delivers substantially better accuracy.

Industry implications: multiplication-free inference

| Config | Stages T | Bits/real param | Storage (LLaMA-2 7B) | |---|---|---|---| | Fairy2i-W1 | 1 | 1-bit | ~1.75 GB | | Fairy2i-W2 | 2 | 2-bit | ~3.5 GB | | FP16 | — | 16-bit | ~14 GB |

With $B_{\text{re}}, B_{\text{im}} \in \{-1, 0, 1\}$ , matrix multiplication reduces to additions/subtractions/skips; multiplying by ±i is just swapping real/imaginary parts with sign flips. Stages run in parallel, giving near O(1) theoretical latency and ~25% FLOPs reduction — a major win for on-device, privacy-preserving inference.

Key takeaways and limitations

1. Representation efficiency beats brute-force compression — complex {±1, ±i} naturally matches the 2-bit space. 2. Mathematical transformations give zero-cost gains — the widely-linear transform is strictly lossless. 3. Reuse beats training from scratch — building on pre-trained checkpoints is far more efficient.

Limitations: 30B-token training is still costly for small labs, dedicated CUDA kernels are not fully developed, hardware acceleration needs real-world testing, and validation on larger models (e.g., LLaMA-3 70B) remains future work.

References

1. Feiyu Wang, Xinyu Tan, Bokai Huang, et al. *Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in {±1, ±i}*. arXiv:2512.02901, 2025 (v3 2026-01-29). 2. Yu You. Paper reading: Fairy2i. WeChat public account, 2026-05-07. 3. Touvron et al. *Llama 2: Open Foundation and Fine-Tuned Chat Models*. arXiv:2307.09288, 2023. 4. Ma et al. *The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits*. BitNet series, 2024. 5. Egiazarian et al. *AQLM: Additive Quantization for Language Models*. 2024.