[论文] The role of locality and weight sharing in convolutional neural networ...

论文概要

研究领域: ML 作者: Francesco Mori, Melanie Weber 发布时间: 2026-10-21 arXiv: 2610.30118

中文摘要

卷积神经网络的成功通常归因于局部性和权重共享。然而，这两个属性分别如何塑造学习过程，以及它们的效果如何依赖自然图像的空间相关性，仍然理解不足。我们在教师-学生模型框架下研究这些问题：卷积教师作用于 P 个输入位置，位置间相关系数为 c。我们比较了通过在线随机梯度下降训练的卷积学生和局部连接学生，并在大特征维度 D 极限下精确推导了描述其动力学的常微分方程。对于不相关的位置（c=0），两种学生遵循相同的学习轨迹，但卷积学生的学习速度恰好快 P 倍。我们还发现局部性影响样本复杂度：卷积核大小与教师匹配的学生可用 O_D(D) 个样本恢复教师，而使用更大卷积核则需要 O_D(D log(D)) 个样本。当 c>0 时，空间相关性会降低权重共享的优势。卷积学生的渐近误差随 c 增长，而局部连接学生的渐近误差与 c 无关，在相同学习率下总体更低。然而，当学习率根据训练时长调整时，局部连接学生在早期表现更好，卷积学生在晚期表现更好，其转变点可由解析预测。

原文摘要

The success of convolutional neural networks is often attributed to locality and weight sharing. However, how these two properties separately shape learning, and how their effects depend on the spatial correlations of natural images, remain poorly understood. We address these questions in a teacher-student model with a convolutional teacher acting on P input positions with correlation c between positions. We compare convolutional and locally connected students trained by online stochastic gradient descent, and derive ordinary differential equations that describe their dynamics exactly in the limit of large feature dimension D. For uncorrelated positions (c = 0), the two students follow the same learning trajectory, with the convolutional student learning exactly P times faster. We also fin...

#论文 #arXiv #ML #小凯