Tapered Language Models: A Free Lunch in Parameter Allocation
> Paper: Tapered Language Models (arXiv:2606.23670) > Institutions: Mila × Cornell University > Core finding: Taper MLP width from front to back (wide early layers, narrow late layers). No extra compute, significantly better performance.
---
A Nine-Year-Old Assumption
Since the Transformer was introduced in 2017, virtually every large language model has followed an unwritten rule: every layer looks identical, parameters distributed evenly across depth.
Transformer, RNN, Mamba, Titans — on the surface very different, but sharing the same skeleton: a stack of L identical blocks, each containing a token-mixing module (attention / recurrence / memory) plus an MLP. The MLP's intermediate width d_ff is identical from layer 1 to layer L.
This "uniform allocation" was never questioned. It is convenient, intuitive, and engineering-friendly. But the Mila and Cornell team asked a disarmingly simple question:
If different layers do different work, why should they receive the same budget?
---
The Experiment: Cutting a 440M Model into Three Thirds
The team carved a 440M-parameter Transformer into three thirds (shallow, middle, deep) and varied the MLP intermediate width of each, keeping total parameters fixed:
| Configuration | Shallow | Middle | Deep | Val PPL | |---|---|---|---|---| | Uniform baseline | standard | standard | standard | 16.28 | | Front-wide / Back-narrow (A) | wider | standard | narrower | 15.96 ✅ | | Front-narrow / Back-wide (B) | narrower | standard | wider | 17.29 ❌ | | Middle-wide (C) | standard | wider | standard | 16.61 ❌ |
The result is unambiguous: pushing parameters forward (wider shallow, narrower deep) wins, beating the uniform baseline by 0.32 perplexity points. Reversing the direction collapses performance by more than 1 point — under the same parameter budget, only the allocation direction differs, yet the gap exceeds 1 PPL.
Widening only the middle also hurts. The conclusion: not every wide spot is good — there is a direction. Stack forward, trim back.
---
The Core Insight: Deep Layers Are "Slacking" — Mathematically, Not Just Rhetorically
Why do deep layers need fewer parameters? Because they are mostly slacking.
The team measured a key quantity via cosine similarity: how much each layer's MLP output aligns with the residual stream entering that layer.
- Cosine similarity ≈ 0: MLP output is nearly orthogonal to the existing residual → it computes new features.
- Cosine similarity ≈ 1: MLP output is nearly aligned with the existing residual → it reinforces existing content.
- Shallow layers: \(\rho\) is low → MLPs are genuinely working, emitting new features.
- Deep layers: \(\rho\) steadily climbs → MLPs increasingly resemble "repeaters," merely nudging the existing residual.
- Validation perplexity: 16.28 → 14.44, a 1.84-point improvement
- WikiText perplexity: improved
- LAMBADA perplexity: improved
- Average accuracy across 8 commonsense-reasoning benchmarks: consistent improvement
- Total parameters unchanged
- Training FLOPs unchanged
- Inference FLOPs unchanged
- Attention head count
- Key-value dimension
- Recurrent state size
- Memory slot count
- Expert count in MoE
- Paper: Tapered Language Models (arXiv:2606.23670)
- Authors: Reza Bayat (Mila), Ali Behrouz (Cornell), Aaron Courville (Mila)
- Core concepts: Tapered Language Models (TLMs), depth-aware capacity allocation, cosine tapering
- Keywords: Transformer architecture, MLP width, residual stream alignment, model efficiency
Measurements across the GPT-2 family are highly consistent:
Pearson correlation between depth and \(\rho\) ranged from 0.49 to 0.71 across model sizes — not noise, but a systematic trend. The deeper the layer, the less need for a wide hidden dimension, because there is little genuinely new to compute.
This explains why front-wide / back-narrow works: spend money (parameters) where new information is genuinely created (shallow), and reclaim it from already-saturated layers (deep). Same total budget, spent wisely.
---
Three Tapering Curves
Based on this insight, the team formalizes Tapered Language Models (TLMs):
> With a fixed total parameter budget, make a parameter-bearing dimension monotonically decrease with depth.
For MLP width, the team evaluates three decay functions:
1. Linear decay
Even descent. No plateau, no sharp transition.
2. Cosine decay ⭐ Best
Both ends have soft plateaus — shallow and deep layers hold steady for a stretch, with a smooth transition in between. Consistently beats linear and sigmoid in experiments.
3. Sigmoid decay
Change concentrates in a narrow middle band; both ends are nearly "frozen." Worst performer — most layers are pinned to either d_start or d_end.
Optimal configuration: cosine + d_start / d_end = 1.5 / 0.5. The shallowest MLP is 1.5× baseline width; the deepest is 0.5× baseline width, with a smooth cosine transition.
---
Cross-Architecture, Cross-Scale Validation
What elevates this from "a curiosity" to "an architectural principle" is generality.
The team tested tapering on four fundamentally different token-mixing architectures:
| Architecture | Core mechanism | Tapering effect | |---|---|---| | Standard Transformer | Softmax self-attention | ✅ PPL and downstream tasks both improve | | Gated Attention | Output-gated attention | ✅ PPL and downstream tasks both improve | | Hope-attention | Self-modifying memory (multi-scale frequencies) | ✅ PPL and downstream tasks both improve | | Titans | Neural long-term memory module | ✅ PPL and downstream tasks both improve |
Three scales (440M, 760M, 1.3B) all show consistent gains. This means tapering's benefit is independent of the token-mixing mechanism — it concerns how parameters are distributed across depth, a model-family-agnostic principle.
---
The Effect: A Truly Free Lunch
Using cosine tapering (1.5/0.5) on a 440M Transformer:
Crucially: zero extra cost.
Only existing resources are reallocated — pulling unnecessary width from deep "slacking" MLPs and subsidizing the shallow MLPs that are actually working. This is not "a bigger model"; it is a smarter model.
---
A Bigger Picture: Depth-Aware Architecture
The real significance of TLMs is not "how much to slim MLPs" but revealing a neglected design axis:
Depth is not equal.
Shallow layers extract new features — token relations, syntax patterns, local semantics. Deep layers refine and integrate — pushing extracted features toward final predictions. Refinement does not need the same raw-processing bandwidth as extraction.
This asymmetry exists in Transformer, RNN, Mamba, and Titans. Depth-aware capacity allocation is a universal design axis, independent of model family.
The paper also notes tapering extends beyond MLP width to:
Any dimension that varies with depth can, in principle, be tapered.
---
Limitations and Open Questions
1. The optimal ratio is not universal. 1.5/0.5 was found at 440M; other scales may differ. The U-curve's minimum point may shift with configuration. 2. Only decoder-only models validated. Encoder-decoder architectures (T5, BART) remain untested. 3. Interaction with training dynamics. Tapering helps during pre-training; does the advantage persist through fine-tuning, especially when downstream tasks require deep-layer reprogramming? 4. Interaction with LayerNorm. To bypass LayerNorm's distortion of direction, the cosine-similarity reference uses the unnormalized residual. This hints at subtle interactions between tapering and normalization strategy.
---
One-Line Summary
> Deep layers in large models are slacking — they tweak the existing residual stream rather than invent new features. Since they do not need that many parameters, why not move parameters from deep to shallow? No extra compute, better results. This is not alchemy; it is a basic architectural principle.
The Mila and Cornell team frames the paper as "a free lever hidden in plain sight." Nine years of unquestioned uniform layer width, overturned by a six-page paper.
---