This post shares the arXiv paper 2608.28564, *Learning between the peaks: sharp asymptotics for kernel ridge regression*, by Lorenzo Rizzi, Arie Wortsman Zurich, and Bruno Loureiro (published 2026-08-28).
Summary
The paper studies kernel ridge regression under anisotropic Gaussian data, where the input covariance decays as a power law with exponent \(\alpha \geq 0\) for polynomial inner-product kernels. The authors derive asymptotically sharp expressions for the kernel spectrum and the generalization error in the polynomial high-dimensional regime \(n = \Theta(d^\kappa)\), revealing how anisotropy reshapes the learning curves.
Weak anisotropy (0 < α < 1)
The problem remains effectively high-dimensional and retains some features of the isotropic case, while departing from it in others:
- The variance still peaks at integer sample complexities \(\kappa \in \mathbb{N}\), but these peaks are progressively damped as \(\alpha\) grows.
- For targets strongly aligned with the data's principal directions, the bias drops at fractional sample complexities, decoupling the bias transition from the interpolation peaks.
- The variance no longer depends on the sample size, plateauing under unregularized interpolation or vanishing at an explicit rate under a fixed ridge penalty.
- The bias undergoes a sharp transition controlled by the target's decay rate: below a threshold, learning is abrupt rather than gradual; above it, the bias decays as a power law, recovering the classical source and capacity rates.
- Paper link: <https://arxiv.org/abs/2608.28564>
Strong anisotropy (α > 1)
The problem has constant effective dimension:
The authors finally specialize these results to single-index targets, showing how the alignment between the index and the data's principal directions determines the impact of anisotropy on learning.
Original Abstract (excerpt)
> We study kernel ridge regression under anisotropic Gaussian data, where the input covariance decays as a power law with exponent \(\alpha\geq 0\) for polynomial inner-product kernels. We derive asymptotically sharp expressions for the kernel spectrum and the generalization error in the polynomial high-dimensional regime \(n=\Theta(d^\kappa)\), revealing how anisotropy reshapes the learning curves. For weak anisotropy (\(0<\alpha<1\)), the problem remains effectively high-dimensional and retains some features of the isotropic case, while departing from it in others: the variance still peaks at integer sample complexities \(\kappa\in\mathbb{N}\), but these peaks are progressively damped as \(\alpha\) grows; meanwhile, for targets strongly aligned with the data's principal directions, the bias drops at fractional sample complexitie...