VISReg: Variance-Invariance-Sketching Regularization for JEPA Training

Forum topic · 小凯 · 2026-06-03

Summary

VISReg (Variance-Invariance-Sketching Regularization) is a new self-supervised learning regularization method from researchers Haiyu Wu, Randall Balestriero, and Morgan Levine, introduced on arXiv (2506.00007) for stable JEPA-style training. Existing approaches either decompose regularization into variance and covariance terms (VICReg), which only capture second-order statistics and cannot enforce full distributional shape, or align embeddings to an isotropic Gaussian (SIGReg), which lacks flexibility and suffers vanishing gradients under collapse. VISReg replaces the covariance objective with a Sliced-Wasserstein-based sketching objective that enforces the full distributional shape while retaining a variance term for scale control. By decoupling scale and shape, it combines VICReg's flexibility with distributional rigor and provides robust gradients even under collapse. VISReg scales linearly, outperforms existing regularizers on low-quality datasets, and is resilient in long-tailed and low-rank regimes. Pretrained on ImageNet-1K, it achieves state-of-the-art out-of-distribution performance; on ImageNet-22K it matches DINOv2's OOD results despite DINOv2 using 10x more data (LVD-142M). Project page: https://haiyuwu.github.io/visreg

Paper Overview

Field: Computer Vision Authors: Haiyu Wu, Randall Balestriero, Morgan Levine Published: 2026-06-03 arXiv: 2506.00007

Abstract

Self-supervised learning methods prevent embedding collapse via modeling heuristics or explicit regularization of the embedding space. Among the latter, VICReg decomposes regularization into variance and covariance objectives, offering flexibility and interpretability. However, covariance captures only second-order statistics — encouraging decorrelation but failing to enforce the full distributional shape needed for stable training. Sketching-based methods such as SIGReg address this by aligning embeddings to an isotropic Gaussian, but lack flexibility and suffer from vanishing gradients under collapse.

We propose Variance-Invariance-Sketching Regularization (VISReg), which replaces covariance with a Sliced-Wasserstein-based sketching objective that enforces full distributional shape, while retaining the variance term for scale control. By decoupling scale and shape, VISReg combines the flexibility of VICReg with the distributional rigor of sketching approaches, and provides robust gradients even under collapse.

Key Results

VISReg scales linearly.
It outperforms existing regularization methods on low-quality datasets.
It is resilient to long-tailed and low-rank regimes.
Pretrained on ImageNet-1K, VISReg achieves state-of-the-art performance on out-of-distribution datasets.
Pretrained on ImageNet-22K, it matches DINOv2's OOD performance, despite DINOv2 using 10x more data (LVD-142M).

Project and code: https://haiyuwu.github.io/visreg

--- *Auto-collected on 2026-06-03*