Paper Overview
Field: Machine Learning Authors: Jin Guo, Roy Y. He, Jean-Michel Morel arXiv: 2506.08634
Abstract
Understanding how training data shape neural network predictions is a central problem in modern learning theory. In 2020, Pedro Domingos proposed an interpolation formula valid for every model learned by deterministic gradient descent. It expresses the model's prediction as an integral, along the optimization path, of a data-dependent kernel that aligns the model's gradients at the test and training data. Such a first-order characterization remains valid for models trained with batch-based stochastic optimization.
In this paper, the authors develop second-order forms of these interpolation formulas. They show that the leading path-kernel interpolation is supplemented by a curvature-weighted interpolation term. For stochastic gradient descent, an additional sampling-induced component appears, coupling prediction curvature with the covariance of mini-batch gradient noise. The characterization is extended to stochastic gradient descent with momentum, where the interpolation structure is preserved but the weights are modified by memory-dependent factors. Furthermore, the paper establishes concentration estimates for terminal predictions, determining the scale of fluctuations around the expected second-order representation.
These results jointly refine the path-kernel interpretation of neural network predictions.
Key Points
- Second-order extensions of path kernel interpolation formulas for gradient-descent-trained models.
- Leading interpolation is complemented by a curvature-weighted term.
- For SGD, a sampling-induced component couples prediction curvature with mini-batch gradient noise covariance.
- Momentum SGD preserves the interpolation structure, with memory-dependent weight modifications.
- Concentration estimates quantify fluctuations of terminal predictions around the expected representation.
*Auto-collected on 2026-06-09. Source: arXiv:2506.08634*