Why Larger Models Learn More: The Jungle Law of Neurons
> Imagine two students in a classroom taught the same ten problems. Both get the first seven right. The last three are hard problems requiring recombination of earlier knowledge. The first student fails—not because he lacks the concepts, but because his working memory is saturated with the answers to the first seven problems, too "loud" to let new information in. The second student processed the first problems with less effort; his old knowledge quietly moved to the background, leaving room for the new.
This classroom analogy captures the core mechanism quantitatively confirmed by Huang et al. (May 2026) in *Why Larger Models Learn More*. Large models learn tasks that smaller ones cannot—not because they are smarter, but because neuronal competition is less brutal.
| Item | Detail | |------|--------| | Paper | Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention | | Authors | Jing Huang, Daniel Wurgaft, Rachit Bansal, Laura Ruis, Naomi Saphra, David Alvarez-Melis, Andrew Kyle Lampinen, Christopher Potts, Ekdeep Singh Lubana | | Institutions | Stanford / Google DeepMind / University of Michigan, et al. | | arXiv ID | 2605.29548 | | Submitted | May 28, 2026 | | Category | cs.LG | | Core finding | Large models' advantage on rare/complex tasks stems from reduced gradient interference: small models' neurons pour resources into frequent simple tasks, whose overly strong gradients overwrite rare-task features; in large models, gradient updates for common tasks flatten once resources are allocated, leaving room for rare tasks to accumulate |
1. The Phenomenon: Big Models Know What Small Ones Don't
Larger models are better specifically on rare and complex tasks: rare linguistic patterns appearing only dozens of times in pretraining corpora, and problems requiring multiple compositional reasoning steps. Small models fail cliff-edge style; large models succeed.
The traditional answer—more parameters fit more complex functions—is a tautology. It explains nothing about the *mechanism*: how are parameters allocated, and why does having more enable learning more complex things? Huang et al. offer a mechanism-level answer.
2. Competition and Interference: A Zero-Sum Game Among Neurons
Picture a network learning two tasks simultaneously. Task A: classify cats vs. dogs (frequent, simple, many samples). Task B: recognize rare ancient Yi script (low-frequency, complex, few samples).
Each training round, both tasks' gradients flow into the same neurons. Task A's gradient is strong; Task B's is weak. When superimposed, Task A's gradient overwrites Task B's—not intentionally, but physically: A's updates dominate the weight-change direction, and B's weak signal gets noised out.
This is gradient interference. Small models suffer it severely; large models much less. The reason is capacity: in small models, neurons are scarce, contested by multiple tasks. Task A's strong gradients occupy most neurons, leaving too few free ones for Task B to build stable feature representations. In large models, Task A takes only what it needs; remaining neurons quietly listen to Task B's faint signal. No conflict.
The paper calls this "data-induced resource competition"—driven purely by data distribution, not architecture or training algorithm differences.
3. Synthetic Experiments: Isolating Scale as the Only Variable
The authors built a mixture dataset of monotonic scaling-curve tasks with precisely controlled properties (sample frequency, pattern complexity, correct answers), then trained models of different sizes with all hyperparameters fixed. Only model size varied.
Results: small models underperform on rare and complex tasks; large models learn them. Then came the paper's most elegant move—measuring cross-task gradient cosine similarity (how conflicting two tasks' gradients are). As predicted: small models show far higher cross-task gradient similarity. Their neurons are fighting. Large models' neurons each mind their own business.
A second metric—feature coverage: large models learn more, more stable rare-task features. In small models, rare-task features either don't exist or are highly fragmented.
4. OLMo Validation: From Synthetic to the Real World
Synthetic results could be artifacts. So Huang et al. validated on the OLMo family, from 4M to 4B parameters—three orders of magnitude. They embedded new tasks differentiated by frequency and complexity into the pretraining pipeline and compared model scales.
Result: consistent. Only larger OLMo models learn low-frequency, high-complexity tasks; larger models' representations contain more rare-task features and lower cross-task gradient interference—matching the synthetic experiments exactly. This promotes gradient interference from a fun synthetic phenomenon to a reproducible finding in large-scale pretraining.
5. Why This Beats "More Parameters = Better"
Say you're training a mid-sized model: common-task performance matches the big models, rare-task performance lags badly. Under the "bigger is better" paradigm, the only fix is adding parameters—with no idea how many, or whether it will even work.
Under the "gradient interference is the bottleneck" framework, the toolbox opens:
- Re-weight rare-task gradients during training so they aren't drowned out
- Curriculum learning: consolidate common features before introducing rare tasks
- Sparse gradient updates: reduce update frequency on high-confidence samples
- Neuron-allocation regularization: encourage uniform allocation
6. Honest Uncertainties
Is gradient interference the only explanation? The paper offers a "data-centric explanation" but explicitly admits it accounts for only part of the story. Function-approximation capacity, optimization landscape changes, and implicit regularization all participate. Gradient interference is an underestimated mechanism, not the only one.
Does it reproduce at larger scale? OLMo covers 4M→4B. What about 4B→400B? Does interference keep declining linearly or saturate? The paper has no data.
Rarity and complexity are confounded. In the OLMo experiments, low frequency and high complexity often co-occur. Is a task unlearned because samples are scarce—or because it is inherently complex? Not fully disentangled.
Tradeoffs of interference mitigation. Does re-weighting rare-task gradients hurt common tasks? What are the costs in compute efficiency and generalization? Not explored.
7. Scale Isn't Capability—Scale Is the Price of Quiet
Large models are not smarter. Large models are just quieter.
Quiet meaning: when a model is large enough, gradients for common tasks no longer roar as they do in small models—once common patterns are learned, their update magnitudes shrink. Only when the noise is suppressed can rare things be heard.
This is a fundamental advance for scaling philosophy. Scaling used to mean "make the model bigger." Huang et al. hint that models might be made "quiet" by other means. You don't need more parameters—you need less gradient interference. If curriculum design, gradient re-weighting, or allocation regularization could achieve the same quiet in a mid-sized model, you might not need to spend tens of millions of dollars on a larger training run.
The paper doesn't answer *how* to make small models quiet. It only establishes that quiet is real, and it is the core driver of large models' superiority. The next step—buying quiet at minimal cost—remains open.
| Item | Detail | |------|--------| | Core contributions | (1) First quantitative evidence that reduced gradient interference, not raw parameter capacity, drives large models' rare-task learning; (2) Synthetic experiments revealing data-induced neuronal resource competition; (3) Replication across OLMo 4M→4B; (4) Moving scaling from statistical regularity to mechanism-level explanation, opening non-scaling intervention space | | Key limitations | Interference is not the only explanation; untested beyond 4B; rarity entangled with complexity; mitigation strategies remain conceptual |
References: 1. Huang et al., "Why Larger Models Learn More", arXiv:2605.29548, 2026. 2. Kaplan et al., "Scaling Laws for Neural Language Models", 2020. 3. Hoffmann et al., "Training Compute-Optimal LLMs", 2022. 4. Hacohen & Weinshall, "On The Power of Curriculum Learning", ICML, 2019. 5. Tirumala et al., "Data Selection via Importance Resampling", 2023.