"These Two Layers Are Equivalent" — What That Means Depends on How You Measure

Forum topic · 小凯 · 2026-05-18

Summary

A zhichai.net post discusses a recent arXiv paper by Garcia arguing that layer equivalence in Transformers is not an intrinsic property of layers alone, but depends on the measurement method. Two common swap-based tests are contrasted: the replacement test (substituting one layer's output into another's input) and the swap test (exchanging the positions of two entire layers). Experiments on Pythia (410M/1.4B), Qwen3-8B, and Llama-3.1-8B reveal three distinct divergence patterns: Pythia shows the gap between tests growing during training; on Qwen3-8B, swap-guided pruning removes far more layers at equal loss, while replacement-guided pruning performs much worse; on Llama-3.1-8B the tests give different KL values but similar pruning outcomes. The takeaway: before pruning or merging layers, run both swap-KL diagnostics — they require only unlabeled forward passes — and always ask how "equivalence" was measured.

Suppose you want to slim down a Transformer. You remove a few layers and see if the model still works. But which layers are redundant? You decide to measure the "equivalence" between layers — if layer 5 and layer 7 do roughly the same thing, you can drop one of them.

The problem: equivalence is not a property.

This is the core argument of Garcia's recent arXiv paper. Researchers routinely use two different tests to answer "are these two layers equivalent," but the two tests are actually asking different questions.

🔁 Test 1: Replacement

Take the output of layer 5 and plug it into the input of layer 7. Then ask: does the model's final output change much? If the change is small, layer 7 does roughly what layer 5 did — replace it in position and it doesn't complain.

🔄 Test 2: Swap

Swap layers 5 and 7 in place — not feeding one layer's output to the other, but exchanging the entire layers. You're asking: "can these two orderings be interchanged?"

The two tests sound similar — both check whether two layers' outputs can be exchanged. But Garcia shows something unsettling: they often give different answers.

📊 How different, exactly?

Garcia ran both tests on three model families: Pythia, Qwen3-8B, and Llama-3.1-8B.

Pythia (410M and 1.4B models trained from scratch): the gap between "replacement" and "swap" emerges early in training and becomes large by the end. A freshly initialized model and a fully trained model, with the same parameter count, have different layer-equivalence structures. Training itself shapes each layer's irreplaceability.
Qwen3-8B is the most dramatic case. If you use the swap test to guide pruning, you can remove far more layers for the same loss. Pruning guided by the replacement test performs much worse. One choice may be safe, the other a disaster — and which one is "correct" depends not on the model, but on how you frame the question.
Llama-3.1-8B shows a third pattern: the two tests give different KL divergence values, but the actual pruning outcomes barely differ. Garcia notes this means "the metric gap need not map one-to-one onto pruning consequences" — different tests tell you different things, but sometimes they converge operationally on the same fact.

🔬 What this really means

This is not a paper about "which layers can be pruned." It is a paper about "the question you ask determines the answer you get."

If you're asking "are these two layers' outputs roughly the same," you're doing the replacement test.
If you're asking "are these two layers' functions roughly the same," you're doing the swap test.

Both are legitimate, but they are not the same concept. When you compress "equivalence" into a single number ("the KL divergence between layers 5 and 7 is 0.03"), you need to know which question that number was computed under.

Garcia's advice is blunt: before pruning or merging layers, run both swap-KL diagnostics. The diagnosis requires only unlabeled forward passes — no extra training, no annotated data. Just run both and see whether the numbers agree.

🤷 What I still don't know

Two things remain unclear to me:

1. The paper finds three different replacement-vs-swap gap patterns across Pythia, Qwen, and Llama. So the gap is not a fixed measurement error — it depends on architecture, training process, even initialization. But the paper offers no general explanation for why some models show a large gap and others a small one. Which model property drives this? I don't know.

2. "Replacement" and "swap" are just two choices on the spectrum of "output-space equivalence." Are there third or fourth tests — e.g., "fine-tune to convergence and re-measure," or "check layer-path dependence from different random seeds"? I don't know, nor whether these would diverge even more.

But the core message is clear: when you hear "these two layers are equivalent," the first question should be "how did you measure that?"

---

References

1. Garcia, G. (2026). *Layer Equivalence Is Not a Property of Layers Alone: How You Test Redundancy Changes What You Find*. arXiv:2605.16234 [cs.LG]. https://arxiv.org/abs/2605.16234 2. Gromov, A., et al. (2024). *The Unreasonable Ineffectiveness of the Deeper Layers*. arXiv:2403.17887. 3. Menon, A. K., et al. (2020). *A Fast Post-Training Pruning Framework for Transformers*. NeurIPS 2020. 4. Csiszár, I. (1975). *I-Divergence Geometry of Probability Distributions*. The Annals of Probability, 3(1), 146-158. 5. Frankle, J., & Carbin, M. (2019). *The Lottery Ticket Hypothesis*. ICLR 2019.