[论文] When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

论文概要

研究领域: cs.LG 作者: Hasan Amin, Wei-Kai Chang, Rajiv Khanna 发布时间: 2026-09-08 arXiv: 2609.09116

中文摘要

归一化使神经网络的大部分实际上具有尺度不变性，在学习率调度与权重衰减通过参数范数相互作用以控制优化器有效步长的过程中，诱导了一个隐藏反馈回路。我们表明这种相互作用受一个精确离散时间定律支配：单个标量量捕获所有调度和衰减驱动，而范数增长诱导一个反向的几何自猝灭效应。这产生了一个清晰边界，干净地将收缩主导和扩张主导的有效学习率区域分开。为理解底层机制，我们提供了完全求解的归一化回归模型的精确分析，其中动力学降维到二维，并表明平衡点是内在不稳定的，这意味着带权重衰减的恒定学习率无法稳定维持内部均衡，而是产生由离散时间Jacobian结构驱动的周期性行为。我们进一步通过统一齐次优化器框架将这种视角扩展到各优化器，揭示自猝灭强度中的结构性二分，为自适应方法在归一化下表现出系统性更弱的稳定化提供第一性原理解释。在动态系统和神经网络（MLP、CNN、GPT2 / MNIST、CIFAR、wikiText、OpenWebText）上，预测的定律高度精确地成立，并支持通过识别出的标量直接控制训练，性能在预测边界处尖锐达到峰值。这些结果将尺度不变优化分离为单一主导量，为现代深度学习中的训练动力学、优化器行为和调度设计提供了精确且可操作的视角。

原文摘要

Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer. We show that this interaction is governed by an exact discrete-time law: a single scalar quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect. This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes. To understand the underlying mechanism, we provide exact analysis of a fully solved normalized regression model where the dynamics reduce to two dimensions and show that the balance point is intrinsically unstable, implying that constant learning rate with weight decay cannot stably maintain an interior equilibrium and instead produces recurrent behavior driven by discrete-time Jacobian structure. We further extend this perspective across optimizers through unified homogeneous-optimizer framework that reveals a structural dichotomy in self-quenching strength, providing a first-principles explanation for why adaptive methods exhibit systematically weaker stabilization under normalization. Across dynamical systems and neural networks (MLP, CNN, GPT2 / MNIST, CIFAR, wikiText, OpenWebText), the predicted law holds with high precision and enables direct control of training via the identified scalar, with performance peaking sharply at the predicted boundary. Together, these results isolate a single governing quantity for scale-invariant optimization, providing a precise and actionable lens on training dynamics, optimizer behavior, and schedule design in modern deep learning.

---

*自动采集于 2026-09-10*

#论文 #arXiv #AI #小凯