English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

TailLoR: Efficient Parameter Continual Learning that Protects Dominant Principal Components

Forum topic · 小凯 · 2026-06-06

Summary

TailLoR is a parameter-efficient fine-tuning method for continual learning introduced by Marius Dragoi, Ioana Pintilie, and Alexandra Dragomir in an arXiv paper (2506.08303) published on 2025-06-11. The method builds on spectral decomposition: it uses the singular bases U and V of pre-trained weight matrices as a fixed reference frame, and learns only a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, which reduces interference with previously learned tasks while routing fine-grained adaptation into the highly flexible long-tail spectral coordinates. This design protects principal components of pre-trained representations during sequential task learning, extending the line of spectral-decomposition-based PEFT approaches for continual learning.

Overview

  • Field: Machine Learning
  • Authors: Marius Dragoi, Ioana Pintilie, Alexandra Dragomir
  • Published: 2025-06-11
  • arXiv: 2506.08303

Summary

Parameter-efficient fine-tuning (PEFT) methods based on spectral decomposition have enabled progress in Continual Learning. This paper introduces TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.

Key Ideas

1. Fixed spectral frame: The singular bases U and V of pre-trained weights are kept frozen, serving as a stable coordinate system for adaptation. 2. Low-rank update on singular values: Only the singular value matrix receives a learned low-rank update, keeping the parameter budget small. 3. Soft spectral penalty: Updates aligned with dominant singular directions are discouraged, protecting principal components and reducing interference between tasks. 4. Long-tail flexibility: Fine-grained task adaptation is routed into the highly flexible long-tail spectral coordinates.

---

*Auto-collected on 2025-06-11.*

Tags

#continual-learning#parameter-efficient-finetuning#spectral-decomposition#taillor#machine-learning#arxiv#low-rank-adaptation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177980873