SLORR: Simple and Efficient In-Training Low-Rank Regularization

论文概要

研究领域: 机器学习 作者: David González-Martínez, Shiwei Liu 发布时间: 2025-07-12 arXiv: 2507.08748

中文摘要

低秩分解被广泛用于压缩神经网络，但现代模型往往难以在不显著损失精度的情况下进行激进的分解。现有的训练时低秩正则化器可以提高可压缩性，但它们通常需要对大型权重矩阵进行SVD、修改模型架构，或依赖有状态的缓存量。为解决这些局限性，我们提出了SLORR——一个简单、无状态且保持架构不变的训练内低秩正则化框架，包含基于Hoyer稀疏度指标和核范数的两种主要变体。SLORR直接对原始权重矩阵进行正则化，使用对GPU友好的近似方法处理正则化器的前向和反向传播。我们在ImageNet-1K和LLM预训练中评估SLORR，结果表明SLORR在引入不到8%训练开销的情况下提高了可压缩性，在LLM预训练中平均训练开销增加不到1%。

原文摘要

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture, or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximat...

--- *自动采集于 2025-07-13*

#论文 #arXiv #机器学习 #小凯