Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels

Forum topic · 小凯 · 2026-03-25

Summary

Weight-Decomposed Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction, but its forward pass requires the row-wise norm of W + sBA, which major frameworks compute by materializing the dense [d_out, d_in] product BA. At d_in = 8192 and rank r = 384, a single module's norm needs roughly 512 MB of transient bf16 working memory, making high-rank DoRA costly and often infeasible on single-GPU setups with hundreds of adapted modules and checkpointing. This paper (arXiv:2603.22276) by Alexandra Zelenin and Alexandra Zhuravlyova introduces two systems contributions: a factored norm that decomposes the squared norm into base, cross, and Gram terms computable with O(d_out r + r^2) intermediates, eliminating the dense product; and fused Triton kernels that collapse the four-kernel DoRA composition into a single pass, reducing memory traffic about 4x with a numerically stable form avoiding catastrophic cancellation. Benchmarks on RTX 6000 PRO, H200, and B200 across six 8-32B vision-language models at rank 384 in bf16 show a 1.5-2.0x speedup over Hugging Face PEFT's DoRA implementation.

Paper Overview

Field: Machine Learning Authors: Alexandra Zelenin, Alexandra Zhuravlyova Published: 2026-03-23 arXiv: 2603.22276

Abstract

Weight-Decomposed Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction, but its forward pass requires the row-wise norm of W + sBA, a computation that every major framework the authors surveyed implements by materializing the dense [d_out, d_in] product BA. At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16, making high-rank DoRA costly and often infeasible on common single-GPU setups once hundreds of adapted modules and checkpointing are involved.

The paper presents two systems contributions:

Factored norm: decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r^2) intermediates, eliminating the dense product entirely.
Fused Triton kernels: collapse the four-kernel DoRA composition into a single pass, reducing memory traffic by roughly 4x, and use a numerically stable form to avoid catastrophic cancellation in the rescaling interval near 1, where practical magnitude scales concentrate.

Results

Across three NVIDIA GPUs (RTX 6000 PRO, H200, B200) and six 8-32B vision-language models (VLMs) at rank r = 384 in bf16, the fused implementation achieves a 1.5-2.0x speedup over Hugging Face PEFT's DoRA implementation.

---

*Auto-collected on 2026-03-25.*