Fairy2i: Complex Low-Bit LLM Quantization with ±1 and ±i Weights

Forum topic · ✨步子哥 · 2026-05-07

Summary

Fairy2i, a paper from Peking University (arXiv:2512.02901), introduces a complex-valued low-bit quantization framework that converts existing real-valued LLaMA checkpoints into complex form using a mathematically lossless widely-linear transformation, then quantizes all weights to the four states {±1, ±i}, perfectly filling the 2-bit encoding space. The method combines three steps: the widely-linear real-to-complex conversion, phase-aware quantization projecting weights onto the fourth roots of unity, and recursive residual quantization. On LLaMA-2 7B, Fairy2i-W2 achieves a C4 perplexity of 7.85 (vs 6.63 FP16) and a 62.00 average on zero-shot tasks, substantially outperforming GPTQ, AQLM, and 1-bit real-valued baselines. Storage drops from ~14GB (FP16) to ~3.5GB, peak memory from 28GB to 7GB, and multiplication-free inference reduces FLOPs by about 25%, enabling efficient on-device deployment. The approach reuses pretrained checkpoints instead of training from scratch, cutting costs compared to prior complex-valued models like iFairy.

Overview

Fairy2i is a low-bit quantization method from Peking University (arXiv:2512.02901) that transforms real-valued LLM checkpoints into complex-valued models whose weights take only four states: {±1, ±i}. These four states exactly fill the 2-bit encoding space, unlike ternary {+1, 0, -1} methods (BitNet 1.58-bit) which waste 25% of it.

The Three-Step Method

1. Widely-linear transformation (lossless real→complex bridge): Any real linear transform y = Rx is exactly equivalent to a complex widely-linear form y = Ux + W·conj(x), with U, W ∈ C^(n×m). This is strict mathematical equivalence, not approximation, and total complex parameters are half the real count.

2. Phase-aware quantization: Weights are projected onto the nearest fourth root of unity via b(w) = argmax over {±1, ±i} of Re(w·conj(s)), with axis-wise scaling for real and imaginary parts.

3. Recursive residual quantization: Weights are expressed as a sum of T low-bit terms, each fitting the previous residual. Ablations show T=2 is the sweet spot: going from T=1 to T=2 gains ~19% average accuracy, while T=3 adds only ~1%.

Key Results (LLaMA-2 7B)

C4 perplexity: Fairy2i-W2 scores 7.85 vs 6.63 for FP16, beating GPTQ (10.61), AQLM (8.54), and Real-Binary (11.75).
Zero-shot average: 62.00 vs 64.72 FP16 (only 4.2% lower), far above Real-Binary's 46.21.
Storage: ~3.5GB at W2 vs ~14GB FP16 (75% savings); peak memory drops from 28GB to 7GB.
Training: 30B RedPajama tokens with AdamW and a WSD scheduler; dual learning-rate decay was critical to reaching best perplexity.

Why It Matters

Multiplication-free inference: With weights in {±1, ±i}, matrix multiplication reduces to additions/subtractions/sign-flips; multiplying by ±i just swaps real/imaginary parts. FLOPs drop ~25% and stages run in parallel.
Reuse over from-scratch: Unlike iFairy, which trains complex models from scratch, Fairy2i converts existing LLaMA checkpoints, cutting training cost dramatically.
Smart encoding beats brute compression: The core insight is that {±1, ±i} natively matches 2-bit storage rather than compressing harder.

Limitations

The 30B-token training cost remains significant for small labs; dedicated CUDA kernels are not fully developed; hardware acceleration gains need real-world measurement; and validation on larger models such as LLaMA-3 70B is still pending.

References

1. Feiyu Wang, Xinyu Tan, Bokai Huang, et al. Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in {±1, ±i}. arXiv:2512.02901, 2025 (v3 2026-01-29). 2. Touvron et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288, 2023. 3. Ma et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet), 2024. 4. Egiazarian et al. AQLM: Additive Quantization for Language Models, 2024.