SLORR: Simple and Efficient In-Training Low-Rank Regularization for Neural Networks

Forum topic · 小凯 · 2026-07-13

Summary

SLORR (Simple Low-Rank Regularization) is a training-time regularization framework introduced by David González-Martínez and Shiwei Liu to make neural networks more amenable to low-rank factorization. While low-rank factorization is widely used for compressing neural networks, modern models typically resist aggressive factorization without significant accuracy loss, and existing regularizers require SVDs of large weight matrices, architectural changes, or stateful cached quantities. SLORR addresses these limitations with a simple, stateless, and architecture-preserving approach, offering two main variants based on the Hoyer sparsity metric and the nuclear norm. It directly regularizes the original weight matrices using GPU-friendly approximations for forward and backward passes. Experiments on ImageNet-1K and LLM pretraining show SLORR improves compressibility with less than 8% training overhead, and under 1% average overhead in LLM pretraining.

Paper Overview

Field: Machine Learning Authors: David González-Martínez, Shiwei Liu Published: 2025-07-12 arXiv: 2507.08748

Abstract

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture, or rely on stateful cached quantities.

To address these limitations, the authors introduce SLORR — a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm.

Key Features

Stateless: No cached quantities are required across training steps.
Architecture-preserving: The model architecture remains unchanged; SLORR directly regularizes the original weight matrices.
GPU-friendly: Uses efficient approximations for the forward and backward passes of the regularizers.

Results

Evaluated on ImageNet-1K classification and LLM pretraining.
SLORR improves compressibility with less than 8% training overhead.
In LLM pretraining, average training overhead increase is less than 1%.