Grokking in LLMs: Inductive Bias as the Key to Sudden Generalization

Core insight: Inductive bias is the central mechanism driving the grokking phenomenon—a dynamic "phase transition" during training. Early on, biases (e.g., implicit bias from large initialization) favor quickly fitting a "memorization solution"; later, other biases (explicit weight decay, or Adam's implicit "Slingshot" bias) dominate, pushing the model toward a simpler, generalizing solution.

Key points

Optimization dynamics: from memorization to generalization

• Early phase: Large initialization induces a "lazy" training regime; the model quickly fits training data.
• Phase transition point: The cumulative effect of weight decay begins to dominate, steering the model toward minimum-norm solutions.
• Late phase: The model switches to a "rich" regime, learning meaningful feature representations and generalizing.
• Lyu et al. (2023–2024) provide rigorous mathematical proof of this two-phase structure: Phase 1 behaves like kernel methods (fitting training data); Phase 2 has weight decay dominating, seeking minimum-norm solutions.

Circuit competition perspective

• Memorization circuits: structurally complex, parameter-redundant, fast zero-error fitting, poor generalization.
• Generalization circuits: compact, computationally efficient, capture latent patterns, generalize well.
• Weight decay implicitly prefers low-rank or sparse representations (a proxy for rank minimization). Grokking's phase transition occurs when the efficient generalization circuit overtakes the memorization circuit in efficiency.

Optimizer-specific bias: Adam's "Slingshot" mechanism

• Thilak et al. (2022) found Adam can exhibit periodic, non-monotonic dynamics late in training. This apparent instability is actually a beneficial implicit bias: the model temporarily gets stuck in a local minimum, "accumulates energy," and is slingshotted into a generalizing region.

Empirical observations

• Adam vs SGD: Adam's non-monotonic dynamics and stronger exploration make grokking easier to trigger; SGD converges smoothly and rarely triggers it.
• Weight decay strength: too low (or none) → memorization, no grokking; moderate → clean phase transition; too high → over-penalization and underfitting.
• Algorithmic tasks (modular addition a + b ≡ c (mod p)): sharp, reproducible phase transitions—an ideal testbed for theory.
• Tabular/regression tasks: real data is noisy and complex; simplicity bias may be insufficient or harmful, requiring higher-complexity functions.
• LLMs: grokking appears local and asynchronous—different capabilities (syntax, factual knowledge, reasoning) may grok at different times on different data subsets. "Emergent abilities" can be viewed as a collection of localized grokking events.

Limitations

• Simplicity bias can over-correct on complex tasks, capping performance.
• The optimal bias is highly task-dependent; no unified theory guides its selection.
• Grokking typically requires extremely long training, making it computationally expensive in practice.

Applications

The article outlines training strategies to accelerate and control grokking by tuning weight decay, optimizer choice, and initialization—using inductive biases deliberately to design models that generalize faster.