SLORR: Simple and Efficient In-Training Low-Rank Regularization for Neural Network Compression

Forum topic · 小凯 · 2026-07-13

Summary

SLORR (arXiv:2507.08748) is a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, proposed by David González-Martínez and Shiwei Liu. Low-rank factorization is widely used to compress neural networks, but modern models often cannot be aggressively factorized without significant accuracy loss. Existing training-time low-rank regularizers typically require SVD of large weight matrices, modify model architecture, or depend on stateful cached quantities. SLORR addresses these limitations by directly regularizing the original weight matrices using GPU-friendly approximations for the forward and backward passes. It is instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. Experiments on ImageNet-1K and LLM pretraining show that SLORR improves compressibility with less than 8% training overhead, and adds less than 1% average training overhead in LLM pretraining.

Paper Overview

Field: Machine Learning Authors: David González-Martínez, Shiwei Liu Published: 2025-07-12 arXiv: 2507.08748

Abstract

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture, or rely on stateful cached quantities. To address these limitations, the authors introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers.

Key Results

Evaluated on ImageNet-1K and LLM pretraining
Improves compressibility with less than 8% training overhead
In LLM pretraining, average training overhead increase is less than 1%

Key Contributions

Stateless: no cached quantities required during training
Architecture-preserving: no model modifications needed
GPU-friendly: efficient approximations for both forward and backward passes
Two variants: based on Hoyer sparsity metric and nuclear norm

---

*Auto-collected on 2025-07-13*