Why Larger Models Learn More: Gradient Interference, Not Just Capacity

Forum topic · 小凯 · 2026-05-31

Summary

This forum post reviews the paper 'Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention' (arXiv:2605.29548) by Huang et al. from Stanford, Google DeepMind, and the University of Michigan. The paper offers a mechanistic explanation for why large language models master rare and complex tasks that smaller models fail: reduced gradient interference. In small models, neurons are over-allocated to frequent, simple tasks; strong gradients from these tasks drown out weak signals from rare ones. Larger models have spare capacity, so once frequent tasks are covered, gradient updates quiet down, leaving room for rare-task features to accumulate. The authors demonstrate this with synthetic multi-task experiments—measuring cross-task gradient cosine similarity and feature coverage—and validate it on OLMo models spanning 4M to 4B parameters. The post highlights practical implications (curriculum design, gradient reweighting, sparsity, allocation regularization) and limitations: gradient interference is not the only mechanism, behavior beyond 4B is untested, and task rarity and complexity remain entangled. Its takeaway: larger models are not smarter—they are quieter, and quietness might be bought more cheaply than scale.

Overview

This post from zhichai.net reviews *Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention* (arXiv:2605.29548, May 2026) by Jing Huang, Daniel Wurgaft, Rachit Bansal, Laura Ruis, Naomi Saphra, David Alvarez-Melis, Andrew Kyle Lampinen, Christopher Potts, and Ekdeep Singh Lubana (Stanford / Google DeepMind / University of Michigan).

The core claim: large models learn tasks that small models cannot—not because they are intrinsically smarter, but because neuronal competition is less brutal. Small models allocate neurons heavily to frequent, simple tasks; the strong gradients from these tasks physically dominate weight updates and drown out weak signals from rare tasks (gradient interference). Large models, once frequent tasks are covered, produce small gradient updates on them, leaving quiet capacity for rare-task features to accumulate. The authors call this "data-induced resource competition": it arises purely from data distribution, not architecture or training algorithm.

Key points

The phenomenon. Large models outperform small models specifically on rare linguistic patterns and multi-step compositional tasks. The traditional answer—"more parameters fit more complex functions"—is circular and offers no mechanism.
Gradient interference as the mechanism. With two tasks sharing neurons (e.g., frequent cat/dog classification vs. a rare script), the frequent task's strong gradients overwrite the rare task's weak ones in small models. Larger models have enough neurons that both tasks can establish stable representations without conflict.
Synthetic experiments. Controlling everything but model size, the authors trained models on mixed datasets of precisely characterized tasks. Small models failed on rare/complex tasks; large models learned them. Cross-task gradient cosine similarity was much higher in small models (their neurons "fight"), and rare-task features in small models were absent or fragmented.
OLMo validation. The finding reproduces on real pretraining: OLMo models from 4M to 4B parameters (three orders of magnitude), with new frequency/complexity-differentiated tasks embedded in training. Only larger OLMo models learned low-frequency, high-complexity tasks, with lower cross-task interference—matching the synthetic results.
Practical leverage. If gradient interference is the bottleneck, interventions become thinkable: reweighting rare-task gradients, curriculum learning (solidify common features first), sparse gradient updates for high-confidence samples, and neuron-allocation regularization. The paper does not claim these work—it makes them *actionable*.

Honest uncertainties (per the post)

Gradient interference is a partial explanation; function-class capacity, optimization landscape, and implicit regularization also play roles.
Behavior from 4B to 400B is unknown—does interference reduction saturate?
In OLMo experiments, rarity and complexity are entangled and not fully disentangled.
The trade-offs of interference mitigation (cost to common tasks, compute, generalization) are unexplored.

The post's closing thesis

> "Large models are not smarter. Large models are quieter."

When a model is large enough, gradients from already-learned frequent patterns stop being loud, and rare signals become audible. This reframes scaling: instead of "make the model bigger," the goal becomes "reduce gradient interference." If curriculum design, gradient reweighting, or allocation regularization can achieve the same quietness in mid-sized models, scale might not be the only way to buy it. The paper establishes that quietness is real and central; how to purchase it cheaply remains open.

References cited in the post: 1. Huang et al., "Why Larger Models Learn More", arXiv:2605.29548, 2026. 2. Kaplan et al., "Scaling Laws for Neural Language Models", 2020. 3. Hoffmann et al., "Training Compute-Optimal LLMs", 2022. 4. Hacohen & Weinshall, "On The Power of Curriculum Learning", ICML, 2019. 5. Tirumala et al., "Data Selection via Importance Resampling", 2023.