Paper Overview
Field: Machine Learning Authors: Lorenzo Rizzi, Arie Wortsman Zurich, Bruno Loureiro arXiv: 2608.28564
Abstract
We study kernel ridge regression under anisotropic Gaussian data, where the input covariance decays as a power law with exponent \(\alpha\geq 0\) for polynomial inner-product kernels. We derive asymptotically sharp expressions for the kernel spectrum and the generalization error in the polynomial high-dimensional regime \(n=\Theta(d^\kappa)\), revealing how anisotropy reshapes the learning curves. For weak anisotropy (\(0<\alpha<1\)), the problem remains effectively high-dimensional and retains some features of the isotropic case, while departing from it in others: the variance still peaks at integer sample complexities \(\kappa\in\mathbb{N}\), but these peaks are progressively damped as \(\alpha\) grows; meanwhile, for targets strongly aligned with the data's principal directions, the bias drops at fractional sample complexities, decoupling bias transitions from interpolation peaks. For strong anisotropy (\(\alpha>1\)), the problem's effective dimension is constant: the variance no longer depends on sample size, plateauing under ridgeless interpolation or vanishing at an explicit rate under a fixed ridge penalty. The bias undergoes sharp transitions controlled by the target's decay rate: below a threshold, learning is abrupt rather than gradual; above it, the bias decays as a power law, recovering classical source and capacity rates. The results are specialized to single-index targets, showing how alignment between the index and the data's principal directions determines anisotropy's impact on learning.
---
*Auto-collected on 2026-09-01 from zhichai.net.*