[论文] SLORR: Simple and Efficient In-Training Low-Rank Regularization

论文概要

研究领域: ML 作者: David González-Martínez, Shiwei Liu 发布时间: 2025-07-12 arXiv: 2507.08748

中文摘要

低秩分解被广泛用于压缩神经网络，但现代模型往往难以在不显著损失精度的情况下进行激进的分解。现有的训练时低秩正则化器可以提高可压缩性，但它们通常需要对大型权重矩阵进行SVD、修改模型架构（引入额外的可训练参数），或依赖有状态的缓存量。为解决这些局限性，我们提出了SLORR——一个简单、无状态且保持架构不变的训练内低秩正则化框架，包含基于Hoyer稀疏度指标和核范数的两种主要变体。SLORR直接对原始权重矩阵进行正则化，使用对GPU友好的近似方法处理正则化器的前向和反向传播，并提供了近似保证。我们首先在ImageNet-1K上评估SLORR，对ResNet-50、ViT-B/16和ViT-L/16进行短周期持续训练，以及对ResNet-18进行预训练，结果表明SLORR在引入不到8%训练开销的情况下提高了可压缩性。我们进一步在135M和560M规模的LLM预训练中评估SLORR-Hoyer：SLORR训练的压缩模型比未正则化模型更好地保持性能，同时平均训练开销增加不到1%。

原文摘要

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the...

--- *自动采集于 2026-07-13*

#论文 #arXiv #ML #小凯