> Wu, Z. et al. *Variable-Width Transformers.* arXiv:2606.18246, 2026. MIT & MIT-IBM Watson AI Lab.
The Question: Does Forcing Models to Be Narrower Make Them Smarter?
The post opens with an anecdote often attributed to Yang Zhilin (Moonshot AI): intelligence is compression — Einstein compressed enormous information into E=mc². The author asks the reverse question: if we force a model to become "narrower," is it forced to become smarter?
MIT's recent work answers: yes, and measurably so.
Uniform Width Is Convenient, Not Optimal
Standard Transformers give every layer the same hidden width (e.g., 2048 across all 24 layers). But layers do different jobs:
- Early layers: surface relations — syntax, collocations, local co-occurrence
- Middle layers: translating low-level fragments into high-level structure
- Late layers: assembling high-level semantics — reasoning, coreference, cross-sentence logic
- Early layers need width: raw tokens are informationally sparse; capacity is needed for local patterns
- Middle layers need narrowness: the low-to-high abstraction jump benefits from a forced compact representation
- Late layers need width: output generation requires expansion capacity
- Wu, Z. et al. (2026). Variable-Width Transformers. *arXiv:2606.18246*. MIT & MIT-IBM Watson AI Lab.
- Tishby, N. & Zaslavsky, N. (2015). Deep Learning and the Information Bottleneck Principle. *IEEE Information Theory Workshop*.
- Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. *arXiv:2001.08361*.
- Ikeda, S. et al. (2025). On the Optimal Allocation of FFN Capacity in Transformers.
If layers do different work, why give them identical capacity?
The Hourglass (X-shaped) Architecture Wins
The team trained V-shaped (narrow-wide), Λ-shaped (wide-narrow), "3"-shaped (narrow-wide-narrow), and X-shaped (wide-narrow-wide) models under a fixed total parameter budget. The X-shape won: a bottleneck at roughly 30% of normal width, positioned around 75% of total depth, with widths tapering down and back up.
Validated on 200M-2B dense models and a 3B MoE model (1B active), results were consistent: ~3% lower perplexity at equal parameters.
| Metric | Uniform baseline | X-shape | Savings | |--------|-----------------|---------|---------| | FLOPs to reach equal loss | 100% | 78.1% | -22% | | Average layer width (KV cache relevant) | 100% | 85.1% | -15% | | 2B perplexity (DCLM) | 2.751 | 2.724 | Lower |
The Deeper Finding: Narrowing Changes What the Model "Thinks"
Comparing residual stream representations, uniform-width models exhibit mid-layer representation collapse — middle-layer representations become overly similar, and discriminability across inputs drops sharply. The X-shaped model avoids this: the bottleneck acts as a mandatory filter, forcing the model to decide which information is worth keeping.
This connects to information bottleneck theory (Tishby et al.): deep networks first fit inputs, then progressively compress representations to what matters for the output. The X-shaped architecture structuralizes this compression — writing the constraint directly into the architecture rather than letting it emerge slowly during training.
> Narrowness forces choice. Choice is the starting point of intelligence.
Why Not V-shaped or Λ-shaped?
Counterintuitively, monotonic shapes both underperform. The authors' original intuition favored the "3"-shape (wider middle, since middle layers do semantic computation). The explanation offered:
The author likens it to an hourglass: the narrow neck doesn't obstruct flow — it regulates it.
Engineering Detail: Keeping the Residual Stream Clean
Different layer widths normally require projection matrices between layers. Instead, the paper uses a fixed global residual dimension (equal to the widest layer). Each layer reads/writes only a subset of dimensions; unused dimensions are simply copied through. Shrinking truncates; widening copies values back from the most recent layer that processed them. No extra parameters, no projection bottleneck — mathematically equivalent to a uniform-width model where some layers are "blind" to certain dimensions.
Scaling Law Implications
Fitting loss-vs-FLOPs curves reveals two findings:
1. The X-shaped curve shifts downward — lower loss at the same FLOPs. 2. The X-shaped curve is steeper — a slightly larger scaling exponent.
If the trend holds, the advantage may widen at 20B+ scale, though this remains an extrapolation. Non-uniform capacity allocation could be the next scaling lever.
MoE Compatibility
On a 3B-total / 1B-active MoE model, the X-shaped variant used 3% fewer active parameters yet still beat the uniform baseline — efficiency gains stack with sparse activation rather than competing with it.
Conclusion: Is Compression Really Intelligence?
The post closes by returning to the pizza analogy: a system that remembers everything without prioritizing is a database; a system forced to decide what to keep within limited capacity exhibits the beginnings of understanding. The X-shaped Transformer's bottleneck is the physical embodiment of that trade-off.