When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

Forum topic · 小凯 · 2026-09-10

Summary

This arXiv paper (2609.09116) by Hasan Amin, Wei-Kai Chang, and Rajiv Khanna analyzes why scale-invariant optimization in normalized neural networks can become unstable. Normalization induces a hidden feedback loop where learning-rate schedules and weight decay interact through the parameter norm to control the optimizer's effective step size. The authors derive an exact discrete-time law governed by a single scalar quantity capturing all schedule and decay forcing, counterbalanced by a geometric self-quenching effect from norm growth. This produces a sharp boundary separating contraction- and expansion-dominated effective learning-rate regimes. Through exact analysis of a fully solved normalized regression model reduced to two dimensions, they show the balance point is intrinsically unstable: constant learning rate with weight decay cannot maintain interior equilibrium, instead yielding recurrent behavior driven by discrete-time Jacobian structure. A unified homogeneous-optimizer framework reveals a structural dichotomy in self-quenching strength, explaining from first principles why adaptive methods show systematically weaker stabilization under normalization. Experiments on dynamical systems and neural networks (MLP, CNN, GPT-2 on MNIST, CIFAR, WikiText, OpenWebText) confirm the law with high precision and enable direct training control, with performance peaking sharply at the predicted boundary.

Paper Overview

Field: cs.LG Authors: Hasan Amin, Wei-Kai Chang, Rajiv Khanna Published: 2026-09-08 arXiv: 2609.09116

English Translation

Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer.

The authors show that this interaction is governed by an exact discrete-time law: a single scalar quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect. This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning-rate regimes.

To understand the underlying mechanism, the paper provides exact analysis of a fully solved normalized regression model where the dynamics reduce to two dimensions. The balance point is intrinsically unstable, implying that a constant learning rate with weight decay cannot stably maintain an interior equilibrium and instead produces recurrent behavior driven by discrete-time Jacobian structure.

The perspective is extended across optimizers through a unified homogeneous-optimizer framework that reveals a structural dichotomy in self-quenching strength, providing a first-principles explanation for why adaptive methods exhibit systematically weaker stabilization under normalization.

Experimental Validation

Across dynamical systems and neural networks (MLP, CNN, GPT-2 / MNIST, CIFAR, WikiText, OpenWebText), the predicted law holds with high precision and enables direct control of training via the identified scalar, with performance peaking sharply at the predicted boundary.

Key Takeaways

A single governing scalar quantity determines stability in scale-invariant optimization
Constant learning rate plus weight decay is intrinsically unstable in normalized networks, producing periodic dynamics
Adaptive optimizers show weaker stabilization under normalization due to a structural dichotomy in self-quenching strength
The theory enables precise, actionable schedule design in modern deep learning