When Do Neural Networks Grok? New Study Predicts AI's 'Eureka Moment' with Power Laws

Forum topic · 小凯 · 2026-09-12

Summary

Grokking is the phenomenon where a neural network first memorizes training data (high training accuracy but random test performance), then suddenly transitions to true generalization after prolonged training—a dramatic shift first reported by DeepMind's Power et al. in 2022. A new paper by Anish Kataria (arXiv:2609.10657) turns this apparent miracle into a predictable science by running 384 networks on modular arithmetic tasks while varying width, data fraction, learning rate, and weight decay. The time to grokking follows a power law: T_grok scales as H^-0.27 * D^-2.04 * eta^-0.50 * lambda^-0.64. Data coverage dominates—doubling data cuts grokking time roughly fourfold, versus only ~20% from doubling model width. A sharp phase boundary at weight decay ~1.0 determines whether grokking occurs at all, and grokking coincides with substantial weight-norm compression (median 42% of memorization-time norm), supporting the view that generalization equals complexity compression. Computationally, scaling width is suboptimal; more diverse data plus strong regularization is the efficient path.

When Do Neural Networks Grok? New Study Predicts AI's 'Eureka Moment'

> *"What I cannot create, I do not understand." — Richard Feynman (post's epigraph, paraphrased)*

This post introduces and explains a recent paper quantifying grokking—the abrupt transition of neural networks from memorization to generalization—and its new scaling law.

What is Grokking?

In 2022, researchers at DeepMind (Power et al., arXiv:2201.02177) trained small transformers on modular arithmetic (e.g., a+b mod p). The network first memorized the training set—training accuracy above 99% while test accuracy stayed at random-guess level. Then, after thousands to hundreds of thousands of additional training steps, test accuracy suddenly jumped to near-perfect generalization. The authors named this phenomenon "grokking," a term borrowed from Robert Heinlein's *Stranger in a Stranger Land*.

For four years, researchers knew grokking *could* happen but couldn't predict *when*.

The New Study

Kataria (2026), *"Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking"* (arXiv:2609.10657), systematically varied four factors across 384 network configurations learning addition mod 113 and division mod 97:

| Factor | Symbol | Values | Meaning | |--------|--------|--------|---------| | Width | H | 128, 256, 512 | Model capacity | | Data fraction | D | 30%, 50%, 70%, 97% | Training coverage | | Learning rate | η | 0.001, 0.003, 0.01, 0.03 | Step size | | Weight decay | λ | 0.1, 0.3, 1.0, 3.0 | Regularization strength |

Key findings

1. A power law for grokking time

T_{grok} \propto H^{-0.27} \cdot D^{-2.04} \cdot \eta^{-0.50} \cdot \lambda^{-0.64}

with R² = 0.732 (0.821 with interaction terms). The hierarchy matters:

Data fraction D (exponent −2.04) dominates. Doubling data cuts grokking time ~4× (2^2.04 ≈ 4.1).
Weight decay (−0.64) and learning rate (−0.50) have moderate effects.
Width (−0.27) matters least: doubling width reduces grokking time only ~20%.

Interpretation: generalization is driven less by model capacity and more by how tightly the data constrains the solution space. The post's takeaway: "more diverse practice beats raw talent."

2. A sharp phase boundary

A critical phase boundary exists at weight decay λ ≈ 1.0:

λ < 1.0: fewer than 60% of runs grok; results are sensitive to learning rate.
λ ≥ 1.0: over 95% of runs grok, regardless of learning rate.

The transition occurs within a 3× range of λ—a true phase transition. Weight decay acts like a "Socratic questioner," making the memorization shortcut too costly and forcing simpler solutions.

3. Grokking is compression

Weight norms decrease monotonically through the transition. At the grokking point, the median weight norm is only 42% of its value at memorization completion (range 0.31–0.54 across configurations). The rate of this compression—not the absolute norm—predicts how fast a network groks. This aligns with Occam's razor: understanding = compression.

4. Computation-optimal strategy

Wider networks need fewer steps but each step costs more, making them less compute-efficient for grokking. The efficient recipe: 1. More, more diverse data (biggest effect) 2. Sufficient regularization (ensures the phase transition happens) 3. Patience—grokking is data-driven, not scale-driven

Broader implications

Memorization is the enemy of generalization early in training; stronger constraints, not bigger models, break the trap.
Grokking is predictable: knowing the training conditions lets you estimate when the transition occurs.
The grokking gap varies enormously: across 297 successful configurations, the interval between memorization and grokking spanned from 100 to over 100,000 steps—a 1000× range.
Understanding is compression: once you grasp core structure, details can be derived—echoing the Feynman technique.

Conclusion

Grokking has moved from a puzzling miracle to a measurable, law-governed phenomenon. When a network memorizes but fails to generalize, the answer may not be more parameters—but more diverse data, stronger regularization, and more time.

References

Kataria, A. (2026). *Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking*. arXiv:2609.10657.
Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177.
Nanda, N., et al. (2023). Progress measures for grokking via mechanistic interpretability. *ICLR*.
Merrill, W., Tsilivis, N., & Shukla, A. (2023). A tale of two circuits: Grokking as competition of sparse and dense subnetworks. arXiv:2303.11873.
Liu, Z., et al. (2023). Omnigrok: Grokking beyond algorithmic tasks.
Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361.