Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Early-Stopped Negative-Shifted Gradient Descent

Forum topic · 小凯 · 2026-07-28

Summary

A new arXiv paper (2607.22474, Peng Zhao, ML) analyzes spectral regularization in overparameterized linear regression. While negative ridge naturally corrects the implicit ridge-like penalty of weak spectral directions, stable negative-ridge endpoints are structurally limited: their pole must remain below the smallest nonzero empirical eigenvalue, and they anti-shrink small eigenvalues more than large ones. The author shows that early-stopped negative-shifted gradient descent escapes this constraint, producing a filter that is smooth at the would-be pole and capable of mixed signs: directions above the ridgeless baseline form a leading prefix, while lower directions are shrunk or exposure-controlled, with the stopping time setting the crossover. In a Gaussian spike-plus-flat model, the paper uncovers a Marchenko-Pastur barrier: the shift cancelling the implicit penalty lies one bulk width above the minimum empirical eigenvalue, and the stopping path improves the risk of every admissible endpoint by a polynomial factor under explicit conditions. The main theorem handles general high-effective-rank tails; localized Duhamel integrals control the non-contractive shift dynamics, and a finite-grid inequality transfers separation to verification in an algorithmic selection procedure.

Paper Overview

Field: Machine Learning Author: Peng Zhao Posted: 2026-07-24 arXiv: 2607.22474

Abstract (translated from the Chinese summary)

In overparameterized linear regression, many weak spectral directions act like a ridge penalty on the signal-bearing spectrum; negative ridge is the natural correction, pushing filters above one. The stable negative-ridge endpoint, however, is structurally limited: its pole must stay below the smallest nonzero empirical eigenvalue, and it anti-shrinks smaller eigenvalues more than larger ones.

Early-stopped negative-shifted gradient descent escapes this constraint. Its filter is smooth at the would-be pole and mixed-sign-capable: above-ridgeless directions form a leading prefix, with lower directions shrunk or exposure-controlled while stopping sets the crossover.

In a Gaussian spike-plus-flat model the author discovers a Marchenko-Pastur barrier: the shift that cancels the implicit penalty lies one bulk width above the smallest empirical eigenvalue, and the stopping path improves the risk of every admissible endpoint by a polynomial factor under explicit conditions.

The main theorem allows general high-effective-rank tails: the trace sets an implicit lower bound, the squared spectrum controls exposure, and a lower-critical path simultaneously recovers all head scales—surpassing positive contraction, and once scales separate, surpassing every uniform rescaling of the ridgeless solution.

Handling non-contractive shift dynamics is the central technical challenge; localized Duhamel integrals control them. A finite-grid inequality preserves separation and transfers it to an algorithmic verification/selection procedure.

Original Abstract (excerpt)

> In overparameterized linear regression, many weak spectral directions act like a ridge penalty on the signal-bearing spectrum; negative ridge is the natural correction, pushing filters above one. The stable negative-ridge endpoint, however, is structurally limited: its pole must stay below the smallest nonzero empirical eigenvalue, and it anti-shrinks smaller eigenvalues more than larger ones. Early-stopped negative-shifted gradient descent escapes this constraint. Its filter is smooth at the would-be pole and mixed-sign-capable: above-ridgeless directions form a leading prefix, with lower directions shrunk or exposure-controlled while stopping sets the crossover. In a Gaussian spike-plus-flat model we discover a Marchenko-Pastur barrier: the shift that cancels the implicit penalty lies a ...

---

*Auto-collected on 2026-07-28.*