LoRA Has Been "Under-Scaled" for Six Years: How One Paper Overturned the α = r Myth

Forum topic · 小凯 · 2026-06-16

Summary

A new paper, "The Hidden Power of Scaling Factor in LoRA Optimization" (Zhang et al., 2026), challenges the long-standing LoRA heuristic of setting the scaling factor α equal to (or half of) the rank r. Through large-scale hyperparameter sweeps across models from 184M to 12B parameters, the authors reveal that α is not interchangeable with the learning rate. While raising the learning rate amplifies both useful signal and bilinear-structure noise, increasing α amplifies pure signal without distorting the optimization landscape. Empirically, the optimal α follows a sub-linear square-root law, α* ≈ C·√r (with C ≫ 1), meaning traditional configs under-scale LoRA by 3–10×. The proposed "LoRA-α" recipe—use α = C·√r and a standard full-finetuning learning rate—improves performance by 3–8% across NLU, NLG, and code tasks, and on a 12B model, r=64 with correct α outperforms traditional r=256. The change is a one-line config tweak, fully backward-compatible with PEFT and Llama-Factory, and reduces dependence on per-task learning-rate tuning.

Paper Reference

Title: *The Hidden Power of Scaling Factor in LoRA Optimization*
Authors: Zicheng Zhang et al. (JD, UCAS, NKU, and collaborators)
Link: https://arxiv.org/abs/2606.12883

---

Key points

1. A counter-intuitive discovery

Users typically configure LoRA with lora_alpha = r/2 or lora_alpha = r, then must raise the learning rate 5–50× to get convergence.
The common explanation ("LoRA has few parameters, so it needs a higher LR") is wrong, the paper argues. The real problem is that α is severely under-scaled.

2. α and learning rate do different things

The LoRA update is W = W₀ + (α/r) · BA, a bilinear structure in low-rank factors B and A.
Learning rate scales both task signal *and* bilinear noise drift (the redundant degrees of freedom).
Scaling factor α scales only the output magnitude of BA, leaving the Hessian geometry intact and introducing no extra drift.
Sweeps across 184M–12B models confirm: as α grows, the optimal learning rate *decreases* and the optimal loss *decreases*. This contradicts six years of community intuition.

3. Overturning the α = r myth

Hu et al. (2021) suggested r=8, α=16 or r=64, α=32; frameworks default to α = r/2 or α = r.
The paper traces this to early empirical choices with no theoretical basis that ossified into "standard practice."
The true scaling law uncovered by large sweeps is:

\alpha^* \approx C \sqrt{r}, \quad C \gg 1

Comparison of conventional vs. optimal α (C typically 10–50):

| r | Conventional α (r/2) | Optimal α (≈ C√r) | Gap | |---|----------------------|-------------------|-----| | 8 | 4 | ~25–40 | 6–10× | | 64 | 32 | ~100–200 | 3–6× | | 256 | 128 | ~300–600 | 2–5× |

The authors label this chronic under-configuration the "Under-Scaling Syndrome."

4. Theoretical explanation: the Signal-Drift framework

Full fine-tuning's Hessian spectrum has a few large eigenvalues (task directions) and many small ones (noise/redundancy).
Low-rank parameterization causes spectral suppression, retaining only an r-dimensional subspace that may be misaligned with the task optimum.
With small α, the effective signal is compressed; users compensate by raising LR, which amplifies bilinear noise drift—a vicious cycle: small α → weak signal → higher LR → more noise → worse convergence → even higher LR.
Increasing α directly amplifies useful signal, keeps the landscape smooth, and allows standard LR values. The paper calls α an "Optimization Accelerator," not a learning-rate substitute.

5. LoRA-α: a zero-cost fix

Change: replace alpha = r with alpha = C * sqrt(r) (C ≈ 10–50), then use a standard full-finetuning LR (e.g., 1e-5 to 5e-5). One-line config tweak.
Compatibility: training and inference pipelines are unchanged; α is absorbed into the merged weight at inference. Works with PEFT, Llama-Factory, etc.
Results:
3–8% average gain over α = r/2 on NLU, NLG, and code tasks.
Matches full fine-tuning on several tasks.
Much more stable across tasks—no per-task LR search.
On a 12B model, LoRA-α with r=64 beats traditional LoRA with r=256, a smaller rank with better results, just by setting α correctly.

6. Practical recommendations

| Model size | Rank r | α (LoRA-α) | Learning rate | vs. traditional α | |------------|--------|-----------|---------------|--------------------| | <1B | 8–16 | 40–80 | 2e-5 | 5–10× | | 7B | 64–128 | 200–400 | 1e-5 | 3–6× | | 13B+ | 128–256 | 400–800 | 5e-6 | 3–5× |

Three immediate actions: (1) raise α (e.g., r=64 → α=128 or 256); (2) drop LR back to the full-FT range; (3) cache a task-specific C = α/√r for reuse.
Caveats: very small r (<8) limits the gain; trivial tasks mask the under-scaling problem; full fine-tuning that already saturates leaves less headroom.

7. Why it matters

LoRA is the de-facto standard for LLM fine-tuning (PEFT, Llama-Factory, industry and academia). The paper exposes a fundamental flaw hidden by six years of "folk wisdom."
The α* ∝ √r law joins the family of scaling laws: effective LoRA capacity depends on both r and α, which reframes debates about how large r needs to be.
Lower reliance on hyperparameter search democratizes fine-tuning for small teams and individual developers without GPU budgets for sweeps.

8. Open question

Does the √r scaling extend to richer LoRA variants such as DoRA (magnitude/direction decomposition) and PiSSA (principal-component initialization)? The paper flags this as the natural next research direction.

---

References

1. Hu, E. J., et al. (2021). *LoRA: Low-Rank Adaptation of Large Language Models*. ICLR 2022. 2. Zhang, Z., et al. (2026). *The Hidden Power of Scaling Factor in LoRA Optimization*. arXiv:2606.12883. 3. Liu, S., et al. (2024). *DoRA: Weight-Decomposed Low-Rank Adaptation*. ICML 2024. 4. Meng, F., et al. (2024). *PiSSA: Principal Singular values and Singular vectors Adaptation*. arXiv preprint.

---

Bottom line: α is not a learning-rate substitute but an independent optimization accelerator. The traditional α = r/2 rule has kept LoRA chronically under-scaled for six years. The true optimum follows α* ∝ √r. Just raise α and use a standard LR, and LoRA can match or even surpass full fine-tuning.