[论文] VISReg: Variance-Invariance-Sketching Regularization for JEPA training

论文概要

研究领域: CV
作者: Haiyu Wu, Randall Balestriero, Morgan Levine
发布时间: 2026-06-03
arXiv: 2506.00007

中文摘要

自监督学习方法通过建模启发式或嵌入空间的显式正则化来防止嵌入坍塌。在后者中，VICReg将正则化分解为方差和协方差目标，提供了灵活性和可解释性。然而，协方差仅捕捉二阶统计量——鼓励去相关但无法强制稳定训练所需的完整分布形状。基于草图的方法（如SIGReg）通过将嵌入对齐到各向同性高斯分布来解决这个问题，但缺乏灵活性，且在坍塌情况下遭受梯度消失。我们提出了方差-不变性-草图正则化（VISReg），用基于Sliced-Wasserstein的草图目标替代协方差，强制完整分布形状，同时保留方差项用于尺度控制。通过解耦尺度和形状，VISReg结合了VICReg的灵活性和草图方法的分布严谨性，即使在坍塌情况下也能提供鲁棒的梯度。我们表明VISReg线性扩展，在低质量数据集上优于现有正则化方法，并且对长尾和低秩机制具有弹性。在ImageNet-1K上预训练，VISReg在分布外数据集上实现了最先进的性能。在ImageNet-22K上预训练，它匹配了DINOv2的OOD性能，尽管后者使用了10倍更多的数据（LVD-142M）。项目与代码：https://haiyuwu.github.io/visreg

原文摘要

Self-supervised learning methods prevent embedding collapse via modeling heuristics or explicit regularization of the embedding space. Among the latter, VICReg decomposes regularization into variance and covariance objectives, offering flexibility and interpretability. However, covariance captures only second-order statistics -- encouraging decorrelation but failing to enforce the full distributional shape needed for stable training. Sketching-based methods such as SIGReg address this by aligning embeddings to an isotropic Gaussian, but lack flexibility and suffer from vanishing gradients under collapse. We propose Variance-Invariance-Sketching Regularization (VISReg), which replaces covariance with a Sliced-Wasserstein-based sketching objective that enforces full distributional shape, whi...

自动采集于 2026-06-03

#论文 #arXiv #CV #小凯