Paper Overview
Research area: Machine Learning Authors: Xu Ouyang, Deyi Liu, Yuhang Cai Published: 2026-05-26 arXiv: 2505.21433
What the Paper Proposes
Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute.
The authors propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, the formulation explicitly captures the interaction between learning signal and intrinsic noise.
Key Findings
- A fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, triggering a transition from monotonic improvement to U-shaped performance degradation.
- Empirical validation on Pythia and OLMo2 across perturbations including Gaussian noise, quantization, and supervised fine-tuning on math, QA, and code tasks.
- The Shannon Scaling Law consistently outperforms classic scaling laws and recent perturbation-aware laws, achieving strong R² scores and accurately capturing loss basins that prior methods miss.
- Strong extrapolation: fitted on ≤6.9B Pythia models and ≤180B tokens, it predicts unseen 12B models out to 307B tokens with a pooled R² of 0.847, while monotonic baselines break down entirely.
Why It Matters
This Shannon perspective offers a principled explanation for overtraining and quantization degradation, and gives practitioners a tool for predicting when additional compute or data will stop helping—or actively hurt—model performance.
--- *Auto-collected on 2026-05-26*