TailLoR: 保护主成分的高效参数持续学习

论文概要

研究领域: ML
作者: Marius Dragoi, Ioana Pintilie, Alexandra Dragomir
发布时间: 2025-06-11
arXiv: 2506.08303

中文摘要

基于谱分解的参数高效微调方法在持续学习中取得了进展。本文提出 TailLoR，利用预训练权重的奇异基 U 和 V 作为固定参考框架，学习应用于奇异值矩阵的低秩更新。软谱惩罚抑制与主导奇异方向对齐的更新，减少干扰，同时将细粒度适应路由到高度灵活的长尾谱坐标。

原文摘要

Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.

自动采集于 2025-06-11

#论文 #arXiv #ML #小凯