[论文] SADL: On the Demand-Supply Geometry of Representation Collapse
研究领域: ML 作者: Junhan Sun, Yigui Feng, Mingzhe Liu, Haoyan Xu, Lei Ye, Ziqi Han, Hanru Bai, Yanzhao Zhang, Hao Zhao, Guofeng Zhang 发布时间: 2026-11-16 arXiv: 2611.2…
论文概要
研究领域: ML 作者: Junhan Sun, Yigui Feng, Mingzhe Liu, Haoyan Xu, Lei Ye, Ziqi Han, Hanru Bai, Yanzhao Zhang, Hao Zhao, Guofeng Zhang 发布时间: 2026-11-16 arXiv: 2611.21596
中文摘要
在联合嵌入预测架构(JEPA)中,约束良好的投影器输出本身并不能防止表征崩溃。我们发现,从初始化开始,编码器表征就被不同输入共享的分量所主导。批量归一化的投影器移除了这个分量,因此在投影器之后的正则化无法纠正它,表征之间的差异会缩小直至崩溃。我们提出 SADL(供需损失),它保留原始训练目标,并在需要时直接约束编码器。SADL 比较投影器输出所需的表征差异与编码器能保留并传递给投影器的差异,仅在供给低于需求时添加对编码器的约束。它与不同的需求正则化器兼容,不引入辅助网络、额外可训练参数或架构变更。在单任务和联合训练的四任务 LeWorldModel 和 INTACT 实验中,SADL 将十个随机种子的崩溃率从 50% 降至 0%。训练一个 epoch 后,编码器有效秩是对应基线的 3.4-6.4 倍,隐变量预测误差降至基线的 1/20-1/303。无需额外调参,SADL 在损失权重跨数量级变化时保持稳定。
原文摘要
In Joint-Embedding Predictive Architectures (JEPAs), a well-constrained projector output does not by itself prevent representation collapse. We find that, from initialization, the encoder representation is dominated by a component shared by different inputs. A batch-normalized projector removes this component, so regularization after the projector cannot correct it, and the differences between representations shrink until they collapse. We propose SADL (Supply And Demand Loss), which keeps the original training objective and constrains the encoder directly when needed. SADL compares the representation differences that the projector output demands with the differences the encoder can retain and pass on to the projector, and adds a constraint on the encoder only when supply falls short of de...
*自动采集于 2026-11-18*
#论文 #arXiv #ML #小凯