You want to slim down a Transformer. Remove a few layers, see if it still runs. But which layers are redundant? You decide to measure the "equivalence" of every layer with every other layer — if layer 5 and layer 7 do roughly the same thing, you can drop one of them.
The problem is: "equivalence" is not a property.
This is the core claim of Garcia's recent arXiv paper. He points out that researchers routinely use two different tests to answer "are these two layers equivalent?" — and the two tests are actually asking different questions.
🔁 Test One: Substitution
You take the output of layer 5 and feed it into layer 7's input position. Then you ask: does the model's final output change much? If it changes little, layer 7's function isn't much different from layer 5's — you replaced it in place and the model didn't complain.
🔄 Test Two: Transposition
You swap layers 5 and 7 — not feeding one layer's output to the other, but exchanging the entire layers. You ask: does the model care whether these two can be reordered?
The two tests sound very similar — both check whether the outputs of two layers can be interchanged. But Garcia shows something unsettling: they often give you different answers.
📊 How Different Exactly?
Garcia ran both tests on three model families: Pythia, Qwen3-8B, and Llama-3.1-8B.
On Pythia — 410M and 1.4B parameter models trained from scratch — the gap between "substitution" and "transposition" appears early in training and becomes large by the end. In other words, a freshly initialized model and a fully trained model, even with identical parameter counts, have different inter-layer equivalence structures. Training itself shapes the layers' irreplaceability.
The Qwen3-8B results are more dramatic. On Qwen, if you use the transposition test to guide pruning, you can prune far more layers at the same loss. Pruning guided by the substitution test performs much worse. One choice may be safe and the other disastrous — and which one is correct doesn't depend on the model itself; it depends on how you phrase the question.
Llama-3.1-8B shows a third pattern: the two tests produce different KL divergence values, but the actual pruning outcomes differ little. Garcia says this means "the measurement gap need not map one-to-one onto pruning consequences" — different tests tell you different things, but sometimes they operationally converge on the same fact.
🔬 What This Really Means
This is not a paper about "which layers can be pruned." It's a paper about "the question you ask determines the answer you get."
If you're asking "are these two layers' outputs about the same," you're using the substitution test. If you're asking "are these two layers' functions about the same," you're using the transposition test. Both are legitimate, but they are not asking the same concept. When you compress "equivalence" into a single number ("the KL divergence between layer 5 and layer 7 is 0.03"), you need to know which question that number was computed to answer.
Garcia's advice is blunt: before pruning or merging layers, run both swap-KL diagnostics. The diagnostics only require unlabeled forward passes — no extra training, no labeled data. Just run both and check whether the numbers agree.
🤷 Things I Don't Know
Two things I couldn't figure out.
First, the paper finds three different "substitution vs. transposition" gap patterns across the Pythia, Qwen, and Llama families. This means the gap is not a fixed measurement error — it depends on architecture, training process, even initialization. But the paper offers no general explanation for "why the gap is large in one model and small in another." Which model property drives this? I don't know.
Second, "substitution" and "transposition" are just two choices on the spectrum of "output-space equivalence." Are there third or fourth tests — e.g., "fine-tune to convergence and then measure the gap," or "check inter-layer path dependence from different random seeds"? I don't know, nor whether these tests would produce even larger disagreements.
But the core message is clear: when you hear "these two layers are equivalent," the first question should be "how did you test it?"
---
References
1. Garcia, G. (2026). *Layer Equivalence Is Not a Property of Layers Alone: How You Test Redundancy Changes What You Find*. arXiv:2605.16234 [cs.LG]. https://arxiv.org/abs/2605.16234
2. Gromov, A., et al. (2024). *The Unreasonable Ineffectiveness of the Deeper Layers*. arXiv:2403.17887.
3. Menon, A. K., et al. (2020). *A Fast Post-Training Pruning Framework for Transformers*. NeurIPS 2020.
4. Csiszár, I. (1975). *I-Divergence Geometry of Probability Distributions*. The Annals of Probability, 3(1), 146-158.
5. Frankle, J., & Carbin, M. (2019). *The Lottery Ticket Hypothesis*. ICLR 2019.