LoRA Has Been 'Under-Scaled' for Six Years: A Paper Overturns the α=r Superstition

Forum topic · 小凯 · 2026-06-16

Summary

A new paper, *The Hidden Power of Scaling Factor in LoRA Optimization* (Zhang et al., arXiv:2606.12883), challenges the long-standing LoRA convention of setting the scaling factor α equal to rank r. Through systematic hyperparameter sweeps across models from 184M to 12B parameters, the authors show that α and the learning rate serve fundamentally different roles: the learning rate amplifies both task-relevant signal and bilinear noise drift, whereas α acts as a pure 'optimization accelerator' that scales the LoRA update without distorting the optimization landscape. They derive an empirical optimal scaling law, α* ≈ C√r with C≫1, demonstrating that conventional α=r/2 leaves LoRA in a chronic 'Under-Scaling Syndrome' that forces users into unusually high learning rates. The proposed LoRA-α recipe, α = C√r combined with standard full-finetuning learning rates, requires only a one-line config change, is backward compatible with PEFT and Llama-Factory, and delivers 3-8% average gains over baseline LoRA, in some cases matching full finetuning or surpassing r=256 LoRA with r=64.

Key points

Core finding: In LoRA, the scaling factor α is not interchangeable with the learning rate. The learning rate amplifies signal *and* bilinear noise, while α amplifies only the effective LoRA update magnitude, behaving as an independent "optimization accelerator."
New scaling law: Systematic sweeps across 184M–12B models reveal α* ≈ C√r with C ≫ 1, contradicting the popular heuristic α = r or α = r/2 from Hu et al. (2021), which the authors say has no theoretical basis and has been perpetuated by community tradition.
Under-Scaling Syndrome: Conventional LoRA configurations drastically under-scale α (by roughly 3–10× depending on rank), forcing practitioners to compensate with 5–50× larger learning rates, which in turn amplifies noise drift and worsens convergence.
Theoretical framing: A signal-drift analysis attributes LoRA's optimization gap to spectral suppression by the low-rank parameterization; larger α directly restores effective signal strength without altering Hessian geometry.
LoRA-α recipe: Replace α = r with α = C·√r (C ≈ 10–50, task-dependent) and reuse the standard full-finetuning learning rate (e.g., 1e-5 – 5e-5). One-line config change, fully backward compatible with PEFT, Llama-Factory, and inference-time weight merging.
Empirical results: Average gains of 3–8% over α = r/2 baselines across NLU, NLG, and code tasks. On a 12B model, r=64 LoRA-α outperforms r=256 conventional LoRA.

Practical takeaways

| Model scale | Rank r | Recommended α (LoRA-α) | Learning rate | vs. traditional α | |---|---|---|---|---| | <1B | 8–16 | 40–80 | ~2e-5 | 5–10× | | 7B | 64–128 | 200–400 | ~1e-5 | 3–6× | | 13B+ | 128–256 | 400–800 | ~5e-6 | 3–5× |

Actionable steps: 1. Increase α (e.g., r=64, α=32 → try α=128–256). 2. Drop the learning rate back to the full-finetuning range once α is enlarged. 3. Pick a task-class constant C = α/√r and reuse it across ranks.

Caveats and open questions

Rank must still be large enough (r ≳ 8); the subspace itself is a bottleneck for very small r.
Trivial tasks may mask the benefit because under-scaling is benign when the loss landscape is easy.
When strong full-finetuning SFT already exists, the headroom for LoRA-α is smaller.
Open question: does the √r scaling generalize to richer LoRA variants like DoRA and PiSSA, which alter the parameterization geometry?

References

Hu, E. J., et al. (2021/2022). *LoRA: Low-Rank Adaptation of Large Language Models*. ICLR 2022.
Zhang, Z., et al. *The Hidden Power of Scaling Factor in LoRA Optimization*. arXiv:2606.12883.
Liu, S., et al. (2024). *DoRA: Weight-Decomposed Low-Rank Adaptation*. ICML 2024.
Meng, F., et al. (2024). *PiSSA: Principal Singular Values and Singular Vectors Adaptation*. arXiv preprint.