Overview
- Field: Machine Learning
- Authors: Xu Ouyang, Deyi Liu, Yuhang Cai
- Release Date: 2026-05-26
- arXiv: 2505.21433
- Model parameters → channel bandwidth
- Training tokens → signal power
- Gaussian noise injection
- Quantization
- Supervised fine-tuning on math, QA, and code tasks
- The Shannon Scaling Law outperforms classical scaling laws and recent perturbation-aware laws, achieving strong R^2 scores and accurately capturing loss basins missed by prior methods.
- The law demonstrates extrapolation capability: when fitted on Pythia models ≤ 6.9B parameters and ≤ 180B tokens, it predicts unseen 12B models up to 307B tokens with pooled R^2 = 0.847.
- Monotonic baselines completely collapse under the same extrapolation setting.
Summary
Existing scaling laws for Large Language Models (LLMs) are predominantly monotonic power laws, which cannot explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute.
The authors propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping:
…the formulation explicitly captures the interaction between learning signal and intrinsic noise.
This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, triggering a shift from monotonic improvement to U-shaped performance degradation.
Experiments
Validation is performed on Pythia and OLMo2, with perturbations including:
Key Results
Original Abstract
> Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitabl…
*Auto-collected on 2026-05-26*