GDIO Deep Dive: Ending Catastrophic Forgetting with Divide-by-Two Math
> Paper: Grow, Don't Overwrite: Fine-tuning Without Forgetting > Authors: Dyah Adila (UW-Madison), Hanna Mazzawi, Benoit Dherin, Xavier Gonzalvo (Google Research) > arXiv: 2603.08647v1 [cs.LG], 9 Mar 2026 > Analysis by: Xiaokai
Key points
- Problem: Standard supervised fine-tuning (SFT) causes catastrophic forgetting — after fine-tuning on translation, original-domain accuracy (e.g., WinoGrande common-sense reasoning) collapses to near zero.
- Method (GDIO): Double the hidden dimension of each MLP block in a function-preserving way:
- Up-projection: duplicate the matrix horizontally —
Ŵ^(1) := [W^(1) | W^(1)] - Down-projection: split it vertically and scale each half by 1/2 —
Ŵ^(2) := [½W^(2); ½W^(2)] - Proof:
[Y | Y] · [½W^(2); ½W^(2)] = ½YW^(2) + ½YW^(2) = YW^(2)— identical output at initialization. Generalizes to any integer k (duplicate k times, scale by 1/k), though k=2 is the empirical sweet spot. - G-Freeze: freeze all original parameters, train only the new ones (~60% of original parameter count). Works well for translation and reasoning tasks.
- G-Train: additionally train the expanded up-projection (original + copies) while freezing all down-projections — motivated by evidence that factual knowledge lives in down-projection layers. Significantly better for complex tasks like MathQA.
- Results: GDIO matches or exceeds SFT on new tasks while nearly perfectly preserving old capabilities; SFT drops to near-zero on original domains.
- Function Vectors: GDIO retains 5/10 causal attention heads and FV cosine similarity of 0.95, vs 2–3 heads and 0.28 for SFT — internal representations barely drift.
- Efficiency: A layer-selection heuristic (fine-tune once, expand the layers with the largest weight updates) means expanding only 9–10 layers achieves full performance, reducing trainable parameters from ~60% to ~30%. Simpler tasks converge with a few expanded layers; complex tasks keep improving as more layers (N up to 20) are expanded, suggesting they need high-rank updates distributed across the network.
- MLP is the sweet spot: Ablations show expanding MLPs beats expanding attention head dimension/count; MLP+attention combined is no better than MLP alone. MLPs are the feed-forward memory storing factual knowledge; attention handles routing.
- Relation to MoE: GDIO resembles a two-expert serial mixture with down-projection acting as the averaging gate — but with no gating network or load balancing needed; it's just copy + scale.
- Limitations (acknowledged by the paper): only MLP expansion (tasks requiring changed attention patterns may suffer); simple layer-selection heuristic; k=2 is an empirical choice; only tested on Gemma-family models.
- Adila, D., Mazzawi, H., Dherin, B., & Gonzalvo, X. (2026). Grow, Don't Overwrite: Fine-tuning Without Forgetting. arXiv:2603.08647v1.
- McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks.
- Kirkpatrick, J., et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS.
- Hu, E. J., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
- Todd, E., et al. (2024). Function Vectors in Large Language Models. arXiv.
Why existing approaches fall short
| Approach | Idea | Problem | |---|---|---| | Regularization (EWC, SI) | Penalize deviation from original weights | Zero-sum trade-off in fixed capacity — resources for remembering are taken from learning | | Network growing (random init) | Insert random new modules | Unstable training; new skills learned from scratch | | Network growing (copy weights) | Duplicate pre-trained blocks | Breaks function preservation — output changes at initialization | | LoRA / PEFT | Low-rank ΔW = ABᵀ on frozen W | Still alters the effective behavior of original weights; targets efficiency, not zero forgetting |
GDIO is the first to satisfy both requirements: it reuses pre-trained knowledge while being function-preserving at initialization — like "taking a new exam while holding a perfect score sheet."
Conceptual notes
Big-picture implications
GDIO signals that *forgetting is not an inherent cost of fine-tuning but an artifact of the fixed-capacity assumption*. Combined with trends like agents that self-extend knowledge and portable skills, it points toward evolvable AI architectures. Commercially, it enables per-client incremental models without degrading base capabilities — and hints at agent systems where "loading a skill" means plugging in new MLP parameters rather than just retrieving documents.
Key quotes
> "A common family of solutions uses regularization... this imposes a zero-sum trade-off: within a fixed-capacity model, any resource allocated to remembering the past is a resource taken away from learning the future."
> "This simple scaling perfectly counteracts the wider internal activation, ensuring the expanded model is mathematically identical to the original at initialization."
> "Our approach achieved a high FV cosine similarity of 0.95 and retained 5 causal attention heads. In contrast, SFT's similarity dropped to 0.28, with only 2–3 heads overlapping."
One-sentence takeaway
GDIO uses simple "copy + divide-by-two" math to make model expansion function-preserving, then trains only the new parameters — achieving 100% new-task capability with ~0% forgetting, where traditional methods fight a doomed zero-sum battle in fixed capacity.