Tapered Language Models: Mila & Cornell Find a "Free Lunch" by Making Parameters "Thinner with Depth"
> Paper: Tapered Language Models (arXiv:2606.23670) > Institutions: Mila × Cornell University > Core finding: Changing MLP width from a "one-size-fits-all" scheme to a "narrower with depth" schedule significantly improves model performance—without spending a single extra FLOP.
---
A Nine-Year-Old Unquestioned Assumption
Since the Transformer's birth in 2017, nearly all large language models have followed an unwritten rule: every layer looks identical, with parameters distributed evenly.
Transformers, RNNs, Mamba, Titans... they differ on the surface but share the same skeleton: a stack of L identical blocks, each with a token-mixing module (attention/recurrence/memory) plus an MLP. The MLP intermediate width d_ff is identical from the first layer to the last.
This "uniform allocation" has never been questioned—it's convenient, intuitive, and engineering-friendly. But the Mila/Cornell team asked a deceptively simple question:
If different layers do different things, why should they receive the same number of parameters?
---
The Experiment: Splitting a 440M Model into Three Sections
The team split a 440M-parameter Transformer into three sections and, under a fixed total parameter budget, assigned different MLP widths to each:
| Configuration | Shallow | Middle | Deep | Val. Perplexity | |------|------|------|------|-----------| | Uniform baseline | Standard | Standard | Standard | 16.28 | | Wide-front / narrow-back (A) | Wider | Standard | Narrower | 15.96 ✅ | | Narrow-front / wide-back (B) | Narrower | Standard | Wider | 17.29 ❌ | | Wide-middle (C) | Standard | Wider | Standard | 16.61 ❌ |
The result is clear: pushing parameters forward (wider shallow layers, narrower deep layers) works best, improving over the uniform baseline by 0.32 perplexity points. The reverse (wider deep layers) collapses, with perplexity rising by more than a full point—under identical parameter budgets, merely changing the "direction of allocation" produces over 1 point of difference.
Widening the middle layers also fails. The effect is directional: pile up front, trim back.
---
Core Insight: Deep Layers Are "Slacking Off"—Mathematically
Why don't deeper layers need as many parameters? Because they are largely coasting.
The team measured a key quantity via cosine similarity: how aligned each layer's MLP output is with the residual stream entering that layer.
- Cosine similarity near 0: the MLP output is nearly orthogonal to the existing residual—it is computing entirely new features.
- Cosine similarity near 1: the MLP output is highly aligned—it is merely reinforcing existing content.
- Shallow layers: low ρ → MLPs do real work, outputting new features.
- Deep layers: ρ climbs steadily → MLPs increasingly act like "repeaters," fine-tuning the existing residual stream.
- Validation perplexity: 16.28 → 14.44, a 1.84-point improvement
- WikiText perplexity: improved
- LAMBADA perplexity: improved
- Average accuracy across 8 commonsense reasoning benchmarks: consistently improved
- Total parameters unchanged
- Training FLOPs unchanged
- Inference FLOPs unchanged
- Attention head count
- Key-value dimension
- Recurrent state size
- Memory slot count
- Expert count in MoE
- Paper: Tapered Language Models (arXiv:2606.23670)
- Authors: Reza Bayat (Mila), Ali Behrouz (Cornell), Aaron Courville (Mila)
- Key concepts: Tapered Language Models (TLMs), depth-aware capacity allocation, cosine tapering
- Keywords: Transformer architecture, MLP width, residual stream alignment, model efficiency
Measurements were highly consistent (holding across the GPT-2 family at different scales):
Pearson correlations ranged from 0.49 to 0.71—a systematic trend, not noise. The deeper the layer, the less a wide MLP is needed, because there is little new left to compute.
This explains why "wide-front, narrow-back" works: you spend your budget (parameters) where new information is genuinely created (shallow layers), and reclaim it from already "saturated" regions (deep layers). Same budget, better spent.
---
Tapered Language Models: Three "Slimming Curves"
Based on this finding, the team proposes Tapered Language Models (TLMs)—an architecture design principle:
> Under a fixed total parameter budget, let certain parameter-carrying dimensions monotonically decrease with layer depth.
For MLP width, a decay function determines each layer's d_ff(l):
1. Linear decay
Steady decline. No plateaus, but no smooth transitions either.
2. Cosine decay ⭐ Best
Soft "plateaus" at both ends—shallow and deep layers each stay stable for a while, with a smooth transition in between. In experiments it consistently beats linear and sigmoid.
3. Sigmoid decay
Change is concentrated in a narrow middle segment; the rest is nearly "frozen." Worst performance—most layers are pinned to either d_start or d_end, failing to exploit intermediate width diversity.
Optimal configuration: cosine tapering with d_start/d_end = 1.5/0.5—the shallowest MLP is 1.5× the baseline width, the deepest 0.5×, with a smooth transition.
---
Cross-Architecture, Cross-Scale Validation: Not a Transformer-Only Trick
What elevates this from "an interesting small finding" to an architectural principle is its generality. Tapering was tested on four distinct token-mixing architectures:
| Architecture | Core mechanism | Tapering effect | |------|---------|-------------| | Standard Transformer | Softmax self-attention | ✅ Improves perplexity and downstream tasks | | Gated Attention | Output-gated attention | ✅ Improves perplexity and downstream tasks | | Hope-attention | Self-modifying memory (multi-layer frequencies) | ✅ Improves perplexity and downstream tasks | | Titans | Neural long-term memory module | ✅ Improves perplexity and downstream tasks |
All three scales (440M, 760M, 1.3B) show consistent results. Tapering's gains are independent of the token-mixing form—they concern how parameters are allocated across depth, an architecture-agnostic principle.
---
Results: A Truly Free Lunch
Cosine tapering (1.5/0.5) on a 440M Transformer:
Crucially: zero extra cost.
You're simply reallocating existing resources—taking unneeded width from deep-layer MLPs that are coasting and subsidizing shallow-layer MLPs doing real work. Not a "bigger model," but a "smarter model."
---
The Bigger Picture: Depth-Aware Architecture Design
The real significance of TLMs isn't "how thin the MLP should be" but a neglected design dimension:
Depth is not equal.
Shallow layers extract new features from raw input—token relations, syntax, local semantics. Deep layers refine and integrate—pushing already-extracted features toward the final prediction. Refinement doesn't require as much "raw-material processing bandwidth" as extraction.
This asymmetry exists not only in Transformers but in RNNs, Mamba, and Titans too, meaning depth-aware capacity allocation is a universal design axis independent of model family.
The paper notes tapering extends beyond MLP width:
Any parameter dimension that varies with depth can, in theory, be tapered.
---
Limitations and Open Questions
1. The optimal ratio isn't universal: 1.5/0.5 was found at the 440M scale; other scales may differ. 2. Only decoder-only models tested: whether encoder-decoder architectures (e.g., T5, BART) benefit is untested. 3. Training dynamics: tapering helps in pre-training, but does the advantage hold during fine-tuning—especially when downstream tasks require deep layers to be "reprogrammed"? 4. Interaction with LayerNorm: to avoid LayerNorm's directional distortion, cosine similarity used unnormalized residuals as reference—hinting at subtle interactions between tapering and normalization strategies.
---
One-Sentence Summary
> Deep layers in large models are "slacking off"—they merely fine-tune the existing residual stream rather than create new features. Since they don't need so many parameters, move them to the shallow layers. Without burning an extra FLOP, performance improves. This isn't an alchemy trick—it's a fundamental principle of architecture design.
The Mila/Cornell team positions the work as "a free lever hidden in plain sight." Nine years of unquestioned uniform layer width, gently overturned by a six-page paper.
---