[论文] Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes an...
论文概要
研究领域: ML
作者: Samet Demir, Zafer Dogan
发布时间: 2025-11-15
arXiv: 2511.13009
中文摘要
我们研究了在多索引教师模型上由随机梯度下降(SGD)训练的两层神经网络,聚焦于与教师子空间正交的第一层参数如何影响泛化。我们将第一层权重分解为与教师对齐的信号分量和正交互补分量,并在比例极限下分析它们与第二层的联合SGD动力学。我们证明,虽然SGD对正交互补分量的更新最初是各向同性的,但训练动力学会在互补分量的Gram矩阵中诱导出依赖于教师的谱尖峰。这种新出现的各向异性为表示方差被放大、泛化性能下降提供了一个具体机制。为了分离这一效应,我们引入分解动力学(Decomposed Dynamics)——一个可解析处理的替代模型,其渐近测试误差动力学与SGD一致。在使用ReLU学生网络的模拟中,冻结正交互补分量的方向比训练它们能改善泛化。最后,我们证明信号分量迅速与第二层权重对齐,揭示了即使在这种浅层设置中也存在强烈的跨层耦合。总之,我们的结果为教师子空间之外的特征学习为何损害泛化提供了一个动力学解释。
原文摘要
We study two-layer neural networks trained by stochastic gradient descent (SGD) on a multi-index teacher model, focusing on how first-layer parameters orthogonal to the teacher subspace affect generalization. We decompose the first-layer weights into a signal component aligned with the teacher and an orthogonal complement, and analyze their joint SGD dynamics together with the second layer in the proportional limit. We show that although SGD updates to the complement are initially isotropic, the training dynamics induce teacher-dependent spectral spikes in the complement Gram matrix. This emergent anisotropy provides a concrete mechanism by which representation variance is amplified, and generalization degrades. To isolate this effect, we introduce Decomposed Dynamics, an analytically trac...
自动采集于 2026-11-20
#论文 #arXiv #ML #小凯