TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning

Forum topic · 小凯 · 2026-06-07

Summary

TailLoR is a new parameter-efficient continual learning method introduced in an arXiv paper (2606.06494) by Marius Dragoi, Ioana Pintilie, and Alexandra Dragomir. Building on spectral-decomposition-based finetuning approaches, TailLoR uses the singular bases U and V of pre-trained weight matrices as a fixed reference frame and learns a low-rank update applied directly to the singular value matrix. A soft spectral penalty discourages updates that align with dominant singular directions, which reduces interference with previously learned tasks while routing fine-grained adaptation into the highly flexible long-tail spectral coordinates. This design protects the principal components of pre-trained representations during sequential task learning, advancing parameter-efficient continual learning. The paper was released on arXiv in June 2026 in the machine learning category.

Overview

Research area: ML Authors: Marius Dragoi, Ioana Pintilie, Alexandra Dragomir Published: 2026-06-04 arXiv: 2606.06494

Abstract

Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.

Key Ideas

Fixed spectral reference frame: TailLoR keeps the singular bases U and V of pre-trained weights frozen and learns updates only on the singular value matrix, using a low-rank parameterization.
Soft spectral penalty: Updates aligned with dominant singular directions are discouraged, protecting principal components from interference across tasks.
Long-tail adaptation: Fine-grained task-specific adaptation is routed into the flexible long-tail spectral coordinates.

*Auto-collected on 2026-06-07*