SLORR: Simple and Efficient In-Training Low-Rank Regularization for Neural Networks

Forum topic · 小凯 · 2026-07-13

Summary

SLORR (arXiv 2507.08748) is a simple, stateless, and architecture-preserving framework for in-training low-rank regularization of neural networks, proposed by David González-Martínez and Shiwei Liu. Unlike existing training-time low-rank regularizers that require SVDs of large weight matrices, architectural modifications with extra trainable parameters, or stateful cached quantities, SLORR directly regularizes the original weight matrices using GPU-friendly approximations with guaranteed approximation quality. It has two main variants based on the Hoyer sparsity metric and the nuclear norm. Experiments on ImageNet-1K with ResNet-50, ViT-B/16, and ViT-L/16 (short continued training) and ResNet-18 (full pretraining) show SLORR improves compressibility with less than 8% training overhead. In LLM pretraining at 135M and 560M parameter scales, SLORR-Hoyer yields compressed models that better preserve performance than unregularized baselines, with average training overhead under 1%.

Paper Overview

Research Area: Machine Learning Authors: David González-Martínez, Shiwei Liu Published: 2025-07-12 arXiv: 2507.08748

Introduction

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they typically:

Require SVDs of large weight matrices
Modify the model architecture (introducing additional trainable parameters)
Rely on stateful cached quantities

The SLORR Framework

SLORR is a simple, stateless, and architecture-preserving framework for in-training low-rank regularization. It is instantiated with two main variants:

SLORR-Hoyer: based on the Hoyer sparsity metric
SLORR-Nuclear: based on the nuclear norm

SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizer, with approximation guarantees.

Results

Image Classification (ImageNet-1K)

Evaluated on ResNet-50, ViT-B/16, and ViT-L/16 with short continued training, as well as ResNet-18 with full pretraining. SLORR improves compressibility while adding less than 8% training overhead.

LLM Pretraining

SLORR-Hoyer was evaluated at 135M and 560M parameter scales. Compressed models trained with SLORR better preserve performance compared to unregularized baselines, with average training overhead under 1%.