From Memorization to Grokking: What Controls a Neural Network's 'Aha Moment'?

> *"If you can't explain it simply, you don't understand it well enough."* — Richard Feynman

What is Grokking?

In 2022, researchers at DeepMind (Alethea Power et al.) discovered a strange phenomenon while training neural networks on modular arithmetic (e.g., "97 + 86 mod 113"). The network first memorized all training data — training accuracy shot above 99% — but performed at chance level on unseen inputs. Then, after thousands to hundreds of thousands of additional training steps, test accuracy suddenly jumped to near-perfect generalization. The network had transitioned from memorization to genuine understanding.

Power et al. named this phenomenon Grokking, a word borrowed from Robert Heinlein's *Stranger in a Strange Land*, meaning a deep, almost intuitive comprehension.

The New Study: Making Grokking Predictable

In September 2026, Anish Kataria published *"Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking"* (arXiv:2609.10657), with the goal of turning grokking from a mystery into a predictable science.

Experimental Setup

Kataria trained 384 neural network configurations on modular arithmetic (addition mod 113, division mod 97), systematically varying four factors:

| Factor | Symbol | Values | Analogy | |--------|--------|--------|---------| | Model width | H | 128, 256, 512 | Working memory capacity | | Data fraction | D | 30%, 50%, 70%, 97% | Coverage of practice material | | Learning rate | η | 0.001, 0.003, 0.01, 0.03 | Speed of learning | | Weight decay | λ | 0.1, 0.3, 1.0, 3.0 | Strictness with mistakes |

For each configuration, he measured the time from memorization (train accuracy > 99%) to grokking (test accuracy > 95%).

Key Findings

1. Grokking time follows a power law

with \(R^2 = 0.732\) (rising to 0.821 with interaction terms). Data coverage dominates: doubling the data cuts grokking time roughly 4x (\(2^{2.04} \approx 4.1\)), while doubling model width cuts it only ~20% (\(2^{0.27} \approx 1.2\)). Generalization depends more on how tightly data constrains the solution space than on model capacity.

2. A sharp phase boundary in weight decay

At λ ≈ 1.0, there is a phase transition: below it, fewer than 60% of configurations grok (with high sensitivity to learning rate); above it, over 95% grok regardless of learning rate. The transition occurs within a 3x change in λ. Weight decay acts like a Socratic questioner, punishing memorization shortcuts and forcing the model toward simpler, more general solutions.

3. Compression accompanies generalization

The weight norm monotonically decreases during the transition from memorization to generalization — the network "loses weight." At the grokking moment, weight norms averaged only 42% of their memorization-phase value (stable across configurations, range 0.31–0.54). This supports the principle that understanding is compression (Occam's razor). Notably, grokking speed correlated with the *rate* of weight-norm decrease, not the absolute norm.

4. Wider networks are not compute-optimal

Although wider networks need fewer steps, each step costs more compute, making them overall less efficient. In the grokking regime, "scale up the model" is not the optimal strategy. The efficient path is:

1. More, more diverse data (largest effect) 2. Moderate regularization (ensures the phase transition occurs) 3. Patience (grokking is data-driven, not model-driven)

What This Means

• Memorization is the enemy of generalization early in training; the escape comes not from bigger models but from stronger constraints (data diversity and regularization).
• Grokking is a phase transition — like water freezing — and is therefore, in principle, predictable.
• The grokking gap varied enormously: among 297 successful configurations, it ranged from 100 to over 100,000 steps (a thousand-fold spread).
• True understanding is compression: a network that groks keeps only the core structure and derives the details.

Conclusion

Four years ago, grokking was a puzzling miracle. Today, a simple power law shows that the "aha moment" follows rules and can be predicted:

\[T_{grok} \propto H^{-0.27} D^{-2.04} \eta^{-0.50} \lambda^{-0.64}\]

Data's power is roughly eight times that of model width (2.04 vs 0.27), regularization opens the door to understanding, and understanding is compression. If a network performs perfectly on training data but poorly on tests, don't rush to add parameters — try more diverse data, stronger regularization, and more time.

References

• Kataria, A. (2026). *Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking*. arXiv:2609.10657.
• Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177.
• Nanda, N., et al. (2023). Progress measures for grokking via mechanistic interpretability. *ICLR*.
• Merrill, W., Tsilivis, N., & Shukla, A. (2023). A tale of two circuits: Grokking as competition of sparse and dense subnetworks. arXiv:2303.11873.
• Liu, Z., et al. (2023). Omnigrok: Grokking beyond algorithmic tasks. arXiv preprint.
• Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361.