English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Skaling Law: Chinchilla and Kaplan Were Each Half Right

Forum topic · ✨步子哥 · 2026-08-10

Summary

A Chinese forum post discusses the Skaling scaling law paper (arXiv:2608.07222) by researchers at FAIR at Meta, which resolves a four-year dispute between Kaplan's law (2020) and Chinchilla's law (2022). The post explains that Chinchilla's additive, independent formulation of loss as a function of parameters N and data D systematically overestimates loss in data-scarce regimes and underestimates it in over-trained regimes. Skaling adds a multiplicative coupling term to Chinchilla's equation, capturing interactions between model size and data volume that independent terms miss. Reported results show 1.5-2x lower MAPE in interpolation settings and 2-3x lower in extrapolation, plus a sparse-grid strategy that reduces compute for fitting scaling laws by roughly 10x. The post highlights that Kaplan and Chinchilla are both limiting cases of Skaling, discusses implications for over-training practices like Llama 3, and notes limitations including the extra coupling parameter, untested generalization across data/architectures, and the lack of a mechanistic explanation for the coupling effect.

Skaling Law: Chinchilla and Kaplan Were Each Half Right

> Paper: https://arxiv.org/abs/2608.07222 > Authors: Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz, Kartik Ahuja (FAIR at Meta)

A Four-Year-Old Dispute

In 2020, OpenAI published Kaplan's law: model loss follows power-law decay with parameter count N, data D, and compute C. The key claim was that model size and data affect loss independently and can be optimized separately.

In 2022, DeepMind published Chinchilla's law: Kaplan was wrong — models and data should scale proportionally, with an optimal token-to-parameter ratio of about 20:1. Chinchilla re-fit Kaplan's data and found the independence assumption led Kaplan to severely underestimate the importance of data.

Since then, Chinchilla has become industry consensus. Open models like Llama, Mistral, and Qwen largely follow the 20:1 ratio.

But one problem has long been overlooked: Chinchilla's prediction error rises sharply in the two extremes — data scarcity and over-training. Using Chinchilla to predict the loss of "10B parameters trained on 1T tokens" overestimates; predicting "1B parameters on 1T tokens" underestimates.

The Skaling paper's first sentence states the problem: "the standard formula systematically under- and over-estimates loss in the data-scarce and over-trained extremes". This is not random noise — it is systematic bias.

Root Cause: The Independence Assumption Is Wrong

Chinchilla's mathematical form is:

\[L(N, D) = L_\infty + \frac{A}{N^\alpha} + \frac{B}{D^\beta}\]

Here N and D appear as two independent terms, each decaying as a power law. That "independence" is the problem.

Skaling's core insight: the effects of model size and data on loss are not independent — they are coupled. How a small model degrades at large data volumes is completely different from how a large model degrades at small data volumes. You cannot describe all (N, D) combinations with a fixed \(A/N^\alpha\) term.

The authors demonstrate this with numerical gradient analysis: on the (N, D) plane, loss contours are not the "independent additive" shape Chinchilla predicts, but clearly curved — the signature of coupling.

Skaling: Adding a Coupling Term

Skaling's mathematical form:

\[L(N, D) = L_\infty + \frac{A}{N^\alpha} + \frac{B}{D^\beta} + \frac{C}{(N^\alpha \cdot D^\beta)^\gamma}\]

Compared to Chinchilla, there is one extra cross term \(\frac{C}{(N^\alpha \cdot D^\beta)^\gamma}\), where \(\gamma\) is the coupling exponent. The term means: when N and D grow simultaneously, an additional synergistic (or offsetting) effect arises that independent terms cannot capture.

Why multiplicative rather than additive coupling? The paper (Section 7.2) argues in detail that additive interaction terms numerically cannot fit the observed contour curvature, while multiplicative coupling can. This echoes Kaplan's original coupled form, but Skaling makes it more precise.

Results: 1.5-3x Error Reduction

Across multiple cross-validation setups, Skaling vs. Chinchilla:

  • Interpolation (predicting points inside the training grid): 1.5-2x lower MAPE
  • Extrapolation (predicting points outside the training grid): 2-3x lower MAPE
  • Sparse grid strategy: with only a few training points from low-compute regions, it can accurately extrapolate the entire grid — saving roughly 10x compute vs. uniform sampling
  • The third point matters most. Training an LLM now costs millions of dollars; if small-scale experiments can accurately predict large-scale performance, enormous trial-and-error costs can be avoided. Skaling's sparse grid strategy makes "predicting 7B-parameter loss from 100M-parameter experiments" feasible — with acceptable error.

    Deeper Implication: Both Kaplan and Chinchilla Were Half Right

    The paper's most elegant insight: Kaplan and Chinchilla are not opposed — they are approximations of Skaling in different limits.

  • As \(\gamma \to 0\) (coupling term vanishes), Skaling reduces to Chinchilla
  • When data far exceeds model capacity needs, the coupling term dominates and Skaling behaves like Kaplan's coupled form
  • It's like relativity reducing to Newtonian mechanics at low speeds — not a matter of right or wrong, but of different domains of applicability. Chinchilla is accurate in the proportional-scaling sweet spot but fails in the data-scarce or over-trained extremes. Skaling unifies both extremes with a single parameter.

    Practical Implications for Training

    1. Over-training finally has theoretical guidance. Llama 3 trains an 8B model on 15T tokens — far beyond Chinchilla's 20:1 (which would be 160B tokens). Such over-training is "suboptimal" under Chinchilla, but under Skaling it can be correctly explained — the coupling term makes the marginal loss of over-training smaller than Chinchilla predicts.

    2. Small-model experiments predicting large-model performance become more reliable. The sparse grid strategy lets teams fit reliable scaling laws with 1/10 the compute — especially valuable for resource-limited research groups.

    3. A new frontier for data allocation. The paper proposes a "Farseer data allocation frontier" — given a total compute budget, how to split it between N and D to minimize loss. Skaling's optimal allocation differs from Chinchilla's, particularly at non-standard ratios.

    An Honest Assessment

    The limitations are also clear:

  • Skaling adds a parameter \(\gamma\), making fitting harder than Chinchilla. With sparse data, the estimate of \(\gamma\) itself may be unstable.
  • Experiments were mainly on FAIR's internal training data. Whether \(\gamma\) is consistent across different data qualities and model architectures needs further validation.
  • The physical meaning of the coupling term remains unclear. Why would N and D interact synergistically? Because larger models extract more from data, or because larger data better activates larger models' capacity? The paper gives numerical evidence but no mechanistic explanation.
Overall, though, this is the most important scaling law work in recent years. It does not "overturn Chinchilla" — it "completes" it, extending a law accurate in the sweet spot into one accurate across the full range. Scientific progress is often not about overturning old theories, but finding their boundaries of applicability and proposing a more general new theory. Skaling is exactly that kind of progress.

---

Paper: Videau, Youbi-Idrissi, Lopez-Paz, Ahuja. *Skaling: Chinchilla's Exponents Meet Kaplan's Coupling*. arXiv:2608.07222, 2026. (FAIR at Meta)

Tags

#scaling-laws#chinchilla#kaplan-law#llm-training#model-scaling#fair-meta#over-training#compute-efficiency

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633312