TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning

Forum topic · 小凯 · 2026-06-07

Summary

TailLoR is a new parameter-efficient finetuning method for continual learning introduced by Marius Dragoi, Ioana Pintilie, and Alexandra Dragomir (arXiv:2606.06494). Building on spectral-decomposition-based finetuning approaches, TailLoR uses the singular bases U and V of pre-trained weight matrices as a fixed reference frame and learns a low-rank update applied directly to the singular value matrix. A key innovation is a soft spectral penalty that discourages updates aligned with the dominant singular directions of the pre-trained weights. This reduces interference with the most important principal components while routing fine-grained task adaptation into the highly flexible, long-tail spectral coordinates. The approach aims to mitigate catastrophic forgetting in continual learning scenarios while keeping the adaptation parameter-efficient. The paper was published on arXiv in June 2026 under the machine learning category.

Paper Overview

Field: Machine Learning Authors: Marius Dragoi, Ioana Pintilie, Alexandra Dragomir Published: 2026-06-04 arXiv: 2606.06494

Abstract

Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper the authors introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.

Key Ideas

Fixed spectral reference frame: The singular bases U and V of pre-trained weights are kept fixed, so only updates to the singular value matrix are learned, keeping the method parameter-efficient.
Soft spectral penalty: Updates aligned with dominant singular directions are penalized, protecting the principal components that encode core pre-trained knowledge.
Long-tail adaptation: Fine-grained task-specific adaptation is directed into the long-tail spectral coordinates, which are more flexible and less likely to interfere with previous tasks.

---

*Auto-collected on 2026-06-07*