Grokking Phenomenon in Neural Network Training
Grokking is a delayed generalization phase transition observed in neural network training. After an initial period of overfitting and memorization, continued training causes the model to shift toward structured understanding, such as algorithmic circuits or trigonometric representations. In large language model (LLM) pretraining, this phenomenon appears as local asynchronous grokking, where specific data subsets or capabilities transition at different times.
Underlying Mechanisms
Recent research highlights several interacting drivers:
- Numerical stability: Issues such as softmax collapse can trigger or accompany the transition.
- Optimization dynamics: Shifts in the optimizer's effective behavior enable the model to escape memorization plateaus.
- Circuit competition: Different internal circuits compete for representation, and the transition reflects the emergence of more generalizable circuits.
- Grokking describes a delayed generalization phase transition where extended training converts memorization into structured understanding.
- In LLMs, it manifests as local asynchronous grokking across data subsets and capabilities.
- Mechanisms involve numerical stability issues (e.g., softmax collapse), optimization dynamics, and competition between internal circuits.
- 2024–2025 research confirms grokking's relevance to real-world LLM pretraining.
- For researchers: Monitor per-subset loss and internal pathway evolution during pretraining as a cheap indicator of generalization progress.
- For practitioners: Moderately extend training duration and strengthen regularization to potentially induce better generalization. Pay attention to numerical precision optimization, such as using the Muon optimizer, to stabilize training and facilitate the transition.
Studies from 2024 to 2025 have deepened the numerical and phase-transition perspectives, providing evidence that grokking occurs not only in toy setups but also in real-world LLMs during pretraining.