LoCO: Rotating Instead of Adding—A New Take on Fine-Tuning Large Models

Forum topic · 小凯 · 2026-05-18

Summary

A forum post on zhichai.net discusses LoCO (Low-rank Compositional Rotation Fine-tuning), a parameter-efficient fine-tuning method accepted to IJCAI 2026 (arXiv:2605.15916) by Nguyen, Choi, and Tong. Unlike LoRA, which adds a low-rank delta to pretrained weight matrices, LoCO applies multiplicative orthogonal rotations to preserve the geometric structure of the pretrained representation space—vector lengths, angles, and inner-product relationships remain intact. The key challenge is cost: a full d×d orthogonal matrix has d² parameters (e.g., ~16.8M for d=4096). LoCO addresses this by constructing rotations from low-rank skew-symmetric matrices via the exponential map, and chaining multiple simple rotations that can be computed in parallel. Reported experiments cover fine-tuning diffusion transformers (DiT) for image generation, vision transformers (ViT), and language model adaptation, where LoCO matches or exceeds LoRA and other orthogonal and non-orthogonal baselines. The author also raises open questions: when geometric preservation actually matters, how tight the 'near-orthogonal' approximation is, and whether LoCO's extra hyperparameters (rotation rank, chain length) justify added tuning cost over LoRA.

You have a pretrained large model and want to adapt it to your task. Two usual options: full fine-tuning—updating billions of parameters—or parameter-efficient fine-tuning (PEFT) like LoRA.

LoRA adds a low-rank incremental update to the weight matrix, like wiring a small bypass channel in parallel. For a 4096×4096 weight matrix, LoRA replaces the full update with a 4096×rank matrix times a rank×4096 matrix, where rank is typically small—4, 8, 16.

This has been hugely successful, nearly the de facto standard for fine-tuning. But Nguyen, Choi, and Tong's recent paper on arXiv (2605.15916), accepted to IJCAI 2026, points out a subtle problem with LoRA.

Addition vs. Multiplication

LoRA is additive: it adds a matrix ΔW to the original weights W, with only a loose relationship to W—the added update need not preserve the geometry of the original feature space.

Picture a world map. The pretrained model's parameters map "countries" to "capitals" in a space where distances are meaningful—the UK and France are close in "country space." An additive update changes all the angles, distances, and orthogonality relations between new and old feature vectors. "London" might get shifted near "Rome."

For many tasks this doesn't matter. But for tasks that depend on the original representation geometry—retrieval, matching, similarity computation—an additive update can unintentionally destroy the fine structure learned during pretraining.

LoCO does something different: multiplication—not just any multiplication, but an orthogonal transformation, i.e., a rotation.

Why Rotation Helps

A rotation preserves vector lengths. It preserves angles between vectors. It preserves the inner-product structure of the entire feature space. If "UK" gets rotated in representation space, its relation to every other country stays the same.

The catch: rotations in high dimensions aren't cheap. A d×d orthogonal matrix has d² parameters—for d=4096 that's ~16.8M, far more than LoRA.

LoCO's trick is to build the rotation from a low-rank skew-symmetric matrix:

> A skew-symmetric matrix satisfies Aᵀ = -A. In 3D, a skew-symmetric matrix encodes a rotation axis—any 3D rotation can be written as exp(θA) with A skew-symmetric. LoCO generalizes this to high dimensions: a low-rank skew-symmetric matrix (only 2×rank degrees of freedom) generates a near-orthogonal transformation.

Even better, they use a compositional rotation chain—several simple (low-rank) rotations composed in sequence can express more complex transformations. They also designed an approximation scheme so all rotations can be computed in parallel rather than sequentially, keeping the actual computational cost manageable.

Results

Experiments cover three stages. On fine-tuning diffusion transformers (DiT), LoCO beats LoRA and other methods on image generation quality. On vision transformer (ViT) adaptation, it matches or exceeds existing orthogonal and non-orthogonal methods. The same holds for language model adaptation.

What I Don't Know

A few open points:

1. When does "preserving geometry" actually matter? LoRA works well on many tasks despite not being isometric. On which downstream tasks does LoCO's geometric-preservation advantage become measurable? The paper's benchmarks may be comprehensive, but the abstract doesn't address this selectivity question.

2. How "near-orthogonal" is the approximation? Is the rotation generated by a low-rank skew-symmetric matrix fully orthogonal, or is there a controlled error bound? The paper mentions "controlled approximation error," but I can't confirm how controlled from the abstract.

3. More hyperparameters. LoCO adds rotation rank and chain length; LoRA already has two (rank and alpha). Is the extra tuning cost worth it for practitioners? No sensitivity analysis is given.

Still, the core idea is clean and powerful: don't add to your representation space—addition changes the geometry. Rotate instead; rotation preserves everything.

---

References

1. Nguyen, A., Choi, J., & Tong, A. (2026). *LoCO: Low-rank Compositional Rotation Fine-tuning*. arXiv:2605.15916 [cs.LG]. https://arxiv.org/abs/2605.15916 (IJCAI 2026) 2. Hu, E. J., et al. (2022). *LoRA: Low-Rank Adaptation of Large Language Models*. ICLR 2022. 3. Aghajanyan, A., et al. (2021). *Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning*. ACL 2021. 4. Li, X., et al. (2024). *SoRA: Singular-Value Decomposed Low-Rank Adaptation*. ICML 2024. 5. Gallier, J. (2011). *Geometric Methods and Applications* (2nd ed.). Springer: Chapters on rotations and the exponential map.