English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Shannon Scaling Law: Modeling LLM Training as Noisy Channel Information Transmission

Forum topic · 小凯 · 2026-05-26

Summary

A new paper on arXiv (2505.21433) by Xu Ouyang, Deyi Liu, and Yuhang Cai proposes the Shannon Scaling Law, a unified framework that treats LLM training as information transmission over a noisy channel, based on the Shannon-Hartley theorem. Model parameters are mapped to channel bandwidth and training tokens to signal power, capturing the interaction between learning signal and intrinsic noise. The theory reveals a fundamental Shannon capacity for LLMs: scaling model size or data without maintaining sufficient signal-to-noise ratio (SNR) amplifies noise, causing a shift from monotonic improvement to U-shaped performance degradation—explaining phenomena like catastrophic overtraining and quantization-induced degradation that classic monotonic power-law scaling laws cannot. Experiments on Pythia and OLMo2 with perturbations including Gaussian noise, quantization, and supervised fine-tuning on math, QA, and code tasks show the Shannon Scaling Law consistently outperforms classic and perturbation-aware scaling laws with strong R² scores, accurately capturing loss basins. It also extrapolates well: fitted on models up to 6.9B parameters and 180B tokens, it predicts unseen 12B models out to 307B tokens with pooled R²=0.847, while monotonic baselines fail.

Paper Overview

Research area: Machine Learning Authors: Xu Ouyang, Deyi Liu, Yuhang Cai Published: 2026-05-26 arXiv: 2505.21433

What the Paper Proposes

Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute.

The authors propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, the formulation explicitly captures the interaction between learning signal and intrinsic noise.

Key Findings

  • A fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, triggering a transition from monotonic improvement to U-shaped performance degradation.
  • Empirical validation on Pythia and OLMo2 across perturbations including Gaussian noise, quantization, and supervised fine-tuning on math, QA, and code tasks.
  • The Shannon Scaling Law consistently outperforms classic scaling laws and recent perturbation-aware laws, achieving strong R² scores and accurately capturing loss basins that prior methods miss.
  • Strong extrapolation: fitted on ≤6.9B Pythia models and ≤180B tokens, it predicts unseen 12B models out to 307B tokens with a pooled R² of 0.847, while monotonic baselines break down entirely.

Why It Matters

This Shannon perspective offers a principled explanation for overtraining and quantization degradation, and gives practitioners a tool for predicting when additional compute or data will stop helping—or actively hurt—model performance.

--- *Auto-collected on 2026-05-26*

Tags

#llm#scaling-laws#shannon-theory#information-theory#overtraining#quantization#model-capacity#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620811