SLORR: Simple and Efficient In-Training Low-Rank Regularization

Forum topic · 小凯 · 2026-07-13

Summary

SLORR (arXiv:2507.08748) is a simple, stateless, and architecture-preserving framework for in-training low-rank regularization of neural networks, developed by David González-Martínez and Shiwei Liu. Unlike existing training-time low-rank regularizers that require SVDs of large weight matrices, modify model architecture with extra trainable parameters, or rely on stateful cached quantities, SLORR directly regularizes the original weight matrices using GPU-friendly approximations with approximation guarantees. It is instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. Experiments on ImageNet-1K with ResNet-50, ViT-B/16, and ViT-L/16 under short continued training, plus ResNet-18 pretraining, show improved compressibility with less than 8% training overhead. SLORR-Hoyer was also evaluated in LLM pretraining at 135M and 560M scale, where compressed models retain performance better than unregularized baselines while adding under 1% average training overhead.

Paper Overview

Field: Machine Learning Authors: David González-Martínez, Shiwei Liu Published: 2025-07-12 arXiv: 2507.08748

Introduction

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss.

Existing training-time low-rank regularizers can improve compressibility, but they have notable drawbacks:

Require SVDs of large weight matrices
Modify the model architecture by introducing additional trainable parameters
Rely on stateful cached quantities

SLORR

SLORR addresses these limitations as a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, with two main variants:

SLORR-Hoyer: based on the Hoyer sparsity metric
SLORR-nuclear: based on the nuclear norm

SLORR directly regularizes the original weight matrices, using GPU-friendly approximations for the forward and backward passes of the regularizer, and provides approximation guarantees.

Results

ImageNet-1K: Short continued training of ResNet-50, ViT-B/16, and ViT-L/16, plus ResNet-18 pretraining — SLORR improves compressibility with less than 8% training overhead.
LLM pretraining (135M and 560M scale): SLORR-Hoyer-trained compressed models maintain performance better than unregularized models, with less than 1% average training overhead increase.

Reference

Full paper: https://arxiv.org/abs/2507.08748