Fairy2i: 2-Bit Complex LLM Quantization with Weights in {±1, ±i} — A Deep Dive

Forum topic · ✨步子哥 · 2026-05-07

Summary

Fairy2i (Peking University, arXiv:2512.02901) is a low-bit quantization method that converts real-valued LLM weights (e.g., LLaMA-2) into the complex domain via a mathematically lossless widely-linear transform, then quantizes every parameter to one of four fourth roots of unity: {±1, ±i}, perfectly filling the 2-bit encoding space. It addresses three long-standing conflicts: extreme low-bit compression, accuracy preservation, and reuse of pretrained checkpoints. Using phase-aware quantization and recursive residual quantization (T=2 stages), Fairy2i-W2 achieves 7.85 C4 perplexity on LLaMA-2 7B (vs 6.63 FP16), outperforming GPTQ (10.61), AQLM (8.54), and real-valued binary baselines (11.75), with zero-shot average accuracy of 62.00 vs 64.72 full precision. Storage drops to ~3.5GB (from ~14GB FP16), and multiplication-free inference reduces FLOPs by ~25%, making on-device 7B inference practical. This post is a detailed Chinese community review translating and analyzing the paper's theorems, ablations, training recipe (30B RedPajama tokens, WSD scheduler), and industry implications for edge deployment.

Overview

Fairy2i is a quantization framework from Peking University (arXiv:2512.02901) that converts real-valued LLM checkpoints into complex-valued models whose weights are all in {±1, ±i} — exactly four states that perfectly occupy the 2-bit code space. This post is a community reading note explaining why this approach is a mathematical rather than purely engineering solution to low-bit LLM compression.

Key points

The encoding insight: Binary {+1, -1} uses 1-bit for 2 states; ternary {+1, 0, -1} wastes 25% of a 2-bit space; complex {±1, ±i} uses all 4 states of 2 bits — no waste.
Lossless real-to-complex bridge (Theorem 1): Any real linear map $\tilde{y} = R\tilde{x}$ can be written exactly in widely-linear form:

y = Ux + W\bar{x}

with $U, W \in \mathbb{C}^{n \times m}$ , giving half the storage of the real parameters:

\begin{cases} \Re U = \frac{1}{2}(R_{11}+R_{22}), & \Im U = \frac{1}{2}(R_{21}-R_{12}) \\ \Re W = \frac{1}{2}(R_{11}-R_{22}), & \Im W = \frac{1}{2}(R_{12}+R_{21}) \end{cases}

This allows direct reuse of LLaMA-style checkpoints instead of training complex models from scratch.

Phase-aware quantization: Each weight is projected to the nearest fourth root of unity via $b(w) = \arg\max_{s \in \{\pm1, \pm i\}} \Re(w\bar{s})$ , with separate real/imaginary scaling factors for dequantization.
Recursive residual quantization: $W_q \approx \sum_{t=0}^{T-1} \tilde{W}^{(t)}$ , each stage fitting the previous residual. Ablations show T=2 is the sweet spot: C4 PPL drops from 11.03 (T=1) to 8.74 (T=2), but T=3 yields only marginal gain (8.38).

Results on LLaMA-2 7B

| Method | Bits | ARC-e | ARC-c | HellaSwag | PIQA | Wino | Avg. | |---|---|---|---|---|---|---|---| | LLaMA-2 FP16 | 16 | 75.59 | 43.17 | 57.06 | 77.91 | 69.85 | 64.72 | | Fairy2i-W2 | 2 | 72.73 | 39.76 | 53.33 | 76.17 | 68.03 | 62.00 | | Real-Binary | 1 | 53.32 | 22.70 | 35.57 | 66.81 | 52.64 | 46.21 |

Fairy2i-W2 reaches 7.85 C4 perplexity vs 6.63 FP16 (18% degradation), beating GPTQ (10.61), AQLM (8.54), and Real-Binary (11.75).
Storage: ~3.5GB for Fairy2i-W2 vs ~14GB FP16 (~1.75GB for W1); peak memory drops from ~28GB to ~7GB.
Learning-rate scheduling matters: double decay (WSD-style) was key to hitting 7.85 PPL. Training used ~30B RedPajama tokens with AdamW.

Multiplication-free inference

With weights in {±1, ±i}, matrix multiplication reduces to add/subtract/skip operations. Multiplying by ±i amounts to swapping real/imaginary parts with sign flips; stages can run in parallel, giving ~25% FLOP reduction and near-O(1) theoretical latency — a strong fit for phones and edge devices.

Comparison with related work

BitNet (1.58-bit ternary): wastes 25% of the 2-bit space; Fairy2i's complex binary uses it fully.
iFairy: requires training complex models from scratch; Fairy2i reuses real checkpoints via the widely-linear transform.
GPTQ / AQLM / QuIP# (PTQ): better out-of-the-box compatibility but notably worse accuracy at extreme low bits; Fairy2i's continued training (QAT-style) pays off.

Takeaways and open questions

1. Representation efficiency beats brute-force compression — smarter encoding, not harsher quantization. 2. Exact mathematical transforms (widely-linear) deliver zero-cost gains. 3. Building on pretrained checkpoints beats from-scratch training.

Remaining limitations: the 30B-token continued-training budget, lack of dedicated CUDA kernels, unverified hardware acceleration in practice, and validation on larger models (e.g., LLaMA-3 70B) still pending.

References

1. Feiyu Wang, Xinyu Tan, Bokai Huang, et al. *Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in {±1, ±i}*. arXiv:2512.02901, 2025 (v3 2026-01-29). 2. Touvron et al. *Llama 2: Open Foundation and Fine-Tuned Chat Models*. arXiv:2307.09288, 2023. 3. Ma et al. *The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits*, 2024. 4. Egiazarian et al. *AQLM: Additive Quantization for Language Models*, 2024.