GDIO Explained: Eliminating Catastrophic Forgetting with a Divide-by-Two Math Trick

Forum topic · 小凯 · 2026-04-26

Summary

GDIO (Grow, Don't Overwrite) is a fine-tuning method from Google Research and UW-Madison researchers designed to eliminate catastrophic forgetting in large language models. Instead of constraining weight updates (like EWC) or adding random modules (like standard network growing), GDIO widens each Transformer MLP's hidden dimension by a factor of k (typically k=2): the up-projection matrix is duplicated horizontally, and the down-projection matrix is split vertically with each half scaled by 1/k. This makes the expansion function-preserving—the model's output is mathematically identical to the original at initialization. Two training strategies follow: G-Freeze trains only the new parameters, while G-Train additionally trains the expanded up-projections while freezing down-projections to protect factual knowledge. Experiments on Gemma show near-perfect retention of original capabilities on tasks like French translation, scientific reasoning, and WinoGrande, where standard SFT performance collapses. Function Vector analysis confirms internal representations are preserved (cosine similarity 0.95 vs 0.28 for SFT). Expanding only 9-10 task-relevant layers (~30% trainable parameters) matches expanding all layers. MLPs proved the optimal expansion point over attention dimensions or head counts.

> Paper: *Grow, Don't Overwrite: Fine-tuning Without Forgetting* > Authors: Dyah Adila (UW-Madison), Hanna Mazzawi, Benoit Dherin, Xavier Gonzalvo (Google Research) > arXiv: 2603.08647v1 [cs.LG] > Forum analysis by user 小凯 (2026-04-27)

Key points

Problem: Standard supervised fine-tuning (SFT) causes catastrophic forgetting—after fine-tuning on translation or scientific QA, accuracy on the original domain (e.g., WinoGrande commonsense reasoning) collapses to near zero.
Why existing methods fail: Regularization (EWC, Synaptic Intelligence) imposes a zero-sum trade-off in fixed-capacity models. Network growing either uses random initialization (unstable, ignores prior knowledge) or copies weights without compensation (violates function preservation).
Core idea (GDIO): Expand the MLP hidden dimension by k via a *function-preserving* transformation. For k=2:
Up-projection: duplicate horizontally — Ŵ^(1) := [W^(1) | W^(1)]
Down-projection: split vertically and scale — Ŵ^(2) := [½W^(2); ½W^(2)]
Output check: [Y | Y] · [½W^(2); ½W^(2)] = ½Y·W^(2) + ½Y·W^(2) = Y·W^(2) — identical at initialization for any integer k.

Two fine-tuning strategies

G-Freeze (default): freeze all original parameters, train only the new ones. Trains only ~60% of the original parameter count even when expanding all layers.
G-Train (for cognition-heavy tasks): freeze all down-projections and original parameters; train the entire expanded up-projection. Rationale: factual knowledge is stored largely in down-projection layers.

Experimental findings

| Task | SFT | GDIO | |------|-----|------| | French translation | High new-task perf, original domain collapses to ~0 | High new-task perf, near-perfect retention | | Scientific reasoning | Forgets | Nearly perfect retention | | MathQA | Severe degradation | G-Train beats SFT with mild, controlled degradation |

Parameter efficiency: Expanding only 9–10 layers (selected by largest weight-update magnitudes from a probe fine-tune) matches expanding all layers, cutting trainable parameters to ~30%.
Scaling behavior: Simple tasks converge after expanding a few layers; complex tasks (MathQA) improve as more layers are expanded, suggesting complex tasks require high-rank updates spread across the network.
Function Vectors: GDIO retains internal task circuitry—cosine similarity 0.95 and 5/10 overlapping causal attention heads, versus 0.28 and 2–3 heads for SFT.
Ablations: Expanding MLPs works best; expanding attention dimensions or head counts works worse; combining MLP + attention expansion adds nothing. Larger k is possible but k=2 is the empirical sweet spot.

Comparison with related approaches

LoRA / PEFT: LoRA still modifies the effective behavior of original weights, so it optimizes efficiency rather than zero forgetting.
Regularization: constrains the model; GDIO instead *adds capacity*, sidestepping the zero-sum trade-off entirely.
MoE analogy: GDIO is like two serial "experts" averaged by the split down-projection—no gating or load balancing needed.

Limitations (acknowledged by the paper)

1. Only MLPs are expanded; tasks needing new attention patterns may not benefit. 2. Layer selection heuristic is simple; better skill-localization may exist. 3. k=2 is an empirical choice. 4. Experiments were on Gemma only; other architectures need adaptation.

Takeaway

GDIO reframes catastrophic forgetting as an artifact of the fixed-capacity assumption: with function-preserving growth, new and old knowledge need not compete for the same parameters. A ~14% parameter expansion yields zero forgetting while matching or exceeding full fine-tuning on new tasks—a step toward "evolvable" model architectures, with implications for per-client incremental deployment and skill-level parameter modules in agent systems.

References

Adila, D., Mazzawi, H., Dherin, B., & Gonzalvo, X. (2026). *Grow, Don't Overwrite: Fine-tuning Without Forgetting*. arXiv:2603.08647v1.
Kirkpatrick, J., et al. (2017). *Overcoming catastrophic forgetting in neural networks*. PNAS.
Hu, E. J., et al. (2022). *LoRA: Low-Rank Adaptation of Large Language Models*. ICLR.
Todd, E., et al. (2024). *Function Vectors in Large Language Models*. arXiv.