Skaling Law: Chinchilla and Kaplan Were Each Half Right
> Paper: https://arxiv.org/abs/2608.07222 > Authors: Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz, Kartik Ahuja (FAIR at Meta)
A Four-Year-Old Dispute
In 2020, OpenAI published Kaplan's law: model loss follows power-law decay with parameter count N, data D, and compute C. The key claim was that model size and data affect loss independently and can be optimized separately.
In 2022, DeepMind published Chinchilla's law: Kaplan was wrong — models and data should scale proportionally, with an optimal token-to-parameter ratio of about 20:1. Chinchilla re-fit Kaplan's data and found the independence assumption led Kaplan to severely underestimate the importance of data.
Since then, Chinchilla has become industry consensus. Open models like Llama, Mistral, and Qwen largely follow the 20:1 ratio.
But one problem has long been overlooked: Chinchilla's prediction error rises sharply in the two extremes — data scarcity and over-training. Using Chinchilla to predict the loss of "10B parameters trained on 1T tokens" overestimates; predicting "1B parameters on 1T tokens" underestimates.
The Skaling paper's first sentence states the problem: "the standard formula systematically under- and over-estimates loss in the data-scarce and over-trained extremes". This is not random noise — it is systematic bias.
Root Cause: The Independence Assumption Is Wrong
Chinchilla's mathematical form is:
Here N and D appear as two independent terms, each decaying as a power law. That "independence" is the problem.
Skaling's core insight: the effects of model size and data on loss are not independent — they are coupled. How a small model degrades at large data volumes is completely different from how a large model degrades at small data volumes. You cannot describe all (N, D) combinations with a fixed \(A/N^\alpha\) term.
The authors demonstrate this with numerical gradient analysis: on the (N, D) plane, loss contours are not the "independent additive" shape Chinchilla predicts, but clearly curved — the signature of coupling.
Skaling: Adding a Coupling Term
Skaling's mathematical form:
Compared to Chinchilla, there is one extra cross term \(\frac{C}{(N^\alpha \cdot D^\beta)^\gamma}\), where \(\gamma\) is the coupling exponent. The term means: when N and D grow simultaneously, an additional synergistic (or offsetting) effect arises that independent terms cannot capture.
Why multiplicative rather than additive coupling? The paper (Section 7.2) argues in detail that additive interaction terms numerically cannot fit the observed contour curvature, while multiplicative coupling can. This echoes Kaplan's original coupled form, but Skaling makes it more precise.
Results: 1.5-3x Error Reduction
Across multiple cross-validation setups, Skaling vs. Chinchilla:
- Interpolation (predicting points inside the training grid): 1.5-2x lower MAPE
- Extrapolation (predicting points outside the training grid): 2-3x lower MAPE
- Sparse grid strategy: with only a few training points from low-compute regions, it can accurately extrapolate the entire grid — saving roughly 10x compute vs. uniform sampling
- As \(\gamma \to 0\) (coupling term vanishes), Skaling reduces to Chinchilla
- When data far exceeds model capacity needs, the coupling term dominates and Skaling behaves like Kaplan's coupled form
- Skaling adds a parameter \(\gamma\), making fitting harder than Chinchilla. With sparse data, the estimate of \(\gamma\) itself may be unstable.
- Experiments were mainly on FAIR's internal training data. Whether \(\gamma\) is consistent across different data qualities and model architectures needs further validation.
- The physical meaning of the coupling term remains unclear. Why would N and D interact synergistically? Because larger models extract more from data, or because larger data better activates larger models' capacity? The paper gives numerical evidence but no mechanistic explanation.
The third point matters most. Training an LLM now costs millions of dollars; if small-scale experiments can accurately predict large-scale performance, enormous trial-and-error costs can be avoided. Skaling's sparse grid strategy makes "predicting 7B-parameter loss from 100M-parameter experiments" feasible — with acceptable error.
Deeper Implication: Both Kaplan and Chinchilla Were Half Right
The paper's most elegant insight: Kaplan and Chinchilla are not opposed — they are approximations of Skaling in different limits.
It's like relativity reducing to Newtonian mechanics at low speeds — not a matter of right or wrong, but of different domains of applicability. Chinchilla is accurate in the proportional-scaling sweet spot but fails in the data-scarce or over-trained extremes. Skaling unifies both extremes with a single parameter.
Practical Implications for Training
1. Over-training finally has theoretical guidance. Llama 3 trains an 8B model on 15T tokens — far beyond Chinchilla's 20:1 (which would be 160B tokens). Such over-training is "suboptimal" under Chinchilla, but under Skaling it can be correctly explained — the coupling term makes the marginal loss of over-training smaller than Chinchilla predicts.
2. Small-model experiments predicting large-model performance become more reliable. The sparse grid strategy lets teams fit reliable scaling laws with 1/10 the compute — especially valuable for resource-limited research groups.
3. A new frontier for data allocation. The paper proposes a "Farseer data allocation frontier" — given a total compute budget, how to split it between N and D to minimize loss. Skaling's optimal allocation differs from Chinchilla's, particularly at non-standard ratios.
An Honest Assessment
The limitations are also clear:
---
Paper: Videau, Youbi-Idrissi, Lopez-Paz, Ahuja. *Skaling: Chinchilla's Exponents Meet Kaplan's Coupling*. arXiv:2608.07222, 2026. (FAIR at Meta)