[论文] Second-Order Path Kernel Interpolation Formulas in Machine Learning

论文概要

研究领域: ML
作者: Jin Guo, Roy Y. He, Jean-Michel Morel
发布时间: 2025-06-11
arXiv: 2506.08634

中文摘要

理解训练数据如何塑造神经网络预测是现代学习理论的核心问题。2020年Pedro Domingos提出了一个适用于确定性梯度下降所学每个模型的插值公式，将模型预测表示为沿优化路径的数据依赖核积分，该核在测试数据与训练数据处对齐模型梯度。这种一阶刻画对基于批量随机优化训练的模型同样有效。本文发展了这些插值公式的二阶形式。我们证明主导路径核插值由曲率加权插值项补充。对于随机梯度下降，出现额外的采样诱导分量，将预测曲率与小批量梯度噪声的协方差耦合。我们还将表征扩展到带动量的随机梯度下降，其中插值结构被保留但权重由与记忆相关的因子修正。此外，我们建立了终端预测 concentration估计，确定了围绕期望二阶表征的波动尺度。这些结果共同细化了对神经网络预测的路径核解释。

原文摘要

Understanding how training data shape neural network predictions is a central problem in modern learning theory. In 2020, Pedro Domingos proposed an interpolation formula valid for every model learned by deterministic gradient descent. It expresses the model's prediction as an integral, along the optimization path, of a data-dependent kernel that aligns the model's gradients at the test and training data. Such a first-order characterization remains valid for models trained with batch-based stochastic optimization. In this paper, we develop second-order forms of these interpolation formulas. We show that the leading path-kernel interpolation is supplemented by a curvature-weighted interpolation term. For stochastic gradient descent, an additional sampling-induced component appears, coupling...

自动采集于 2026-06-09

#论文 #arXiv #ML #小凯