TailLoR: Parameter-Efficient Continual Learning That Protects Dominant Principal Components

Forum topic · 小凯 · 2026-06-06

Summary

TailLoR is a parameter-efficient fine-tuning method for continual learning introduced by Marius Dragoi, Ioana Pintilie, and Alexandra Dragomir in an arXiv paper (2506.08303, June 2025). Building on spectral decomposition approaches, TailLoR uses the singular bases U and V of pre-trained weight matrices as a fixed reference frame, learning low-rank updates applied to the singular value matrix instead of modifying the bases themselves. A soft spectral penalty discourages updates aligned with the dominant singular directions of the pre-trained weights, which reduces interference between successive tasks in continual learning. At the same time, adaptation is routed into the highly flexible long-tail spectral coordinates, enabling fine-grained task-specific adjustments while preserving the principal components that encode general knowledge. This design allows the model to accumulate new capabilities over sequential tasks with minimal forgetting and a small number of trainable parameters. The paper was aggregated on zhichai.net on 2025-06-11. Original source: https://arxiv.org/abs/2506.08303

Paper Overview

Field: Machine Learning Authors: Marius Dragoi, Ioana Pintilie, Alexandra Dragomir Published: 2025-06-11 arXiv: 2506.08303

Abstract

Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. This paper introduces TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix.

A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.

Key Ideas

Fixed spectral frame: the singular bases U and V from pre-trained weights are kept frozen and used as a shared reference frame across tasks.
Low-rank updates on singular values: adaptation is applied as a low-rank update to the singular value matrix, keeping the parameter footprint small.
Soft spectral penalty: updates aligned with dominant singular directions are penalized, reducing interference between tasks.
Long-tail flexibility: fine-grained, task-specific adaptation is routed into the long-tail spectral coordinates, which are more flexible for new knowledge.

---

*Auto-collected on 2025-06-11.*