You have a pretrained large model and want to adapt it to your task. Two options: full fine-tuning (updating billions of parameters), or parameter-efficient fine-tuning such as LoRA.
LoRA adds a low-rank incremental update to a weight matrix — like wiring a small bypass channel in parallel beside the original matrix. For a 4096×4096 weight matrix, LoRA replaces the full update with a 4096×rank matrix plus a rank×4096 matrix, where rank is typically tiny — 4, 8, 16.
This has been hugely successful and is now the de facto standard for fine-tuning. But Nguyen, Choi, and Tong recently posted a paper on arXiv (2605.15916), accepted to IJCAI 2026 — and they point out a subtle problem with LoRA.
⚠️ Addition vs. Multiplication
LoRA does addition: it adds a matrix ΔW to the original weights W. The relationship between ΔW and W is very loose — what you add doesn't necessarily preserve the geometric structure of the original feature space.
Let me explain with an analogy. Imagine a world map. The pretrained model's parameters map "countries" to "capitals," and in this space distance is meaningful — the UK and France are close in "country space." If you apply an additive update (as LoRA does), the angles, distances, and orthogonality relations between new and original feature vectors all change. "London" might get translated to a spot near "Rome."
For many tasks this may not matter. But for tasks that depend on the original representation geometry — retrieval, matching, similarity computation — an additive update can inadvertently destroy the fine structure learned during pretraining.
LoCO does something different. It multiplies — and not just any multiplication, but an orthogonal transformation. In other words: a rotation.
🔄 Why Rotation Is Better
A rotation preserves vector lengths. A rotation preserves angles between vectors. A rotation preserves the entire inner-product structure of the feature space. If you rotate the position of "UK" in representation space, its relations to all other countries remain intact. You won't accidentally move "UK" next to "Rome."
But the catch: rotation in high-dimensional space isn't cheap. A d×d orthogonal matrix has d² parameters — for d=4096, that's 16 million parameters, far more than LoRA. That would defeat the purpose.
LoCO's trick is to construct the rotation from a low-rank skew-symmetric matrix.
> A skew-symmetric matrix satisfies Aᵀ = -A. In 3D space, a skew-symmetric matrix can encode a rotation axis — in fact, any rotation you see in 3D can be written as exp(θA) where A is skew-symmetric. LoCO generalizes this to high dimensions: a low-rank skew-symmetric matrix (only 2×rank degrees of freedom) generates a near-orthogonal transformation.
Even better, they use a compositional chain of rotations — not one rotation, but several rotations in series. Each rotation is simple (low-rank), but chained together they can express more complex transformations. They also designed an approximate computation scheme so all rotations can be computed in parallel — not sequentially, but all at once. This keeps the actual computational cost manageable.
📊 How Well Does It Work
Experiments cover three settings. On fine-tuning diffusion transformers (DiT), LoCO beats LoRA and other methods on image generation quality. On adapting vision transformers (ViT), it maintains or surpasses existing orthogonal and non-orthogonal methods. The same holds for language model adaptation.
🤷 What I Don't Know
A few things remain unclear.
First, under what conditions does "preserving geometric structure" actually matter? LoRA performs well on many tasks despite not being strictly isometric. On what types of downstream tasks does LoCO's geometry-preserving advantage become a measurable win? The paper's benchmark suite may be comprehensive, but the abstract doesn't address this selectivity question.
Second, how "approximate" is "approximately orthogonal"? Are the rotations generated by low-rank skew-symmetric matrices exactly orthogonal, or is there a controllable error bound? The paper mentions "controlled approximation error," but I can't confirm from the abstract how controlled it is.
Third, LoCO adds extra hyperparameters: the rank of each rotation and the length of the rotation chain. LoRA already has two (rank and alpha). LoCO has more, so tuning costs are higher. For practitioners, is the extra tuning cost worth it? The paper gives no sensitivity analysis.
But the core idea is clean and powerful: don't add things in your representation space — addition changes the geometry. Rotate instead; rotation preserves everything.
---
References
1. Nguyen, A., Choi, J., & Tong, A. (2026). *LoCO: Low-rank Compositional Rotation Fine-tuning*. arXiv:2605.15916 [cs.LG]. https://arxiv.org/abs/2605.15916 (IJCAI 2026)
2. Hu, E. J., et al. (2022). *LoRA: Low-Rank Adaptation of Large Language Models*. ICLR 2022.
3. Aghajanyan, A., et al. (2021). *Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning*. ACL 2021.
4. Li, X., et al. (2024). *SoRA: Singular-Value Decomposed Low-Rank Adaptation*. ICML 2024.
5. Gallier, J. (2011). *Geometric Methods and Applications* (2nd ed.). Springer: Chapters on rotations and the exponential map.