What Survives Distillation? When GPT-2 Is Compressed in Half, Some Circuits Die and Others Learn to Move

arXiv: 2610.37157 — What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2

---

A Question on the Operating Table

Imagine you are a neurosurgeon facing a delicate task: compressing a brain from 12 layers of cortex down to 6. The surgery is a success — the patient speaks fluently afterward, grammar is largely intact, and perplexity (the language model's routine "checkup metric") is only slightly worse. Just as you are about to declare the operation a success, a question occurs to you:

What about the deeper thinking circuits? Are they still okay?

This is exactly what Nabeelah Faumi and Deshan Sumanathilaka did in this paper. They put GPT-2 (a 12-layer Transformer) and its distilled student DistilGPT-2 (6 layers) on the operating table — not with coarse checkup metrics like perplexity, but with six different "microscopes": linear CKA, logit lens, attention pattern decomposition, sparse autoencoder (SAE) feature statistics, and activation patching, comparing circuit by circuit.

They found a chilling phenomenon, which they call the "compression-collapse dichotomy."

Two Circuits, Two Fates

The paper focuses on two of the most thoroughly studied GPT-2 circuits in the mechanistic interpretability community:

Circuit 1: IOI (Indirect Object Identification)

Task: When processing sentences like "When Alice and Bob walked into the room, Alice gave Bob a book," the model needs to know "who gave to whom."

Structure: A multi-stage pipeline spanning all 12 layers — name-mover head → S-inhibition head → induction head. Every link is indispensable, like dominoes running from layer 0 to layer 11.

Fate after distillation: Complete collapse.

• GPT-2 accuracy: 94.8%
• DistilGPT-2 accuracy: 35.2% (nearly chance)
• Name-mover head retention: zero

When you cut 12 layers to 6, the pipeline's physical length is no longer sufficient. The induction head still barely works, but the downstream name-movers are all gone — the pipeline breaks in the middle, and the final link's output never reaches its destination. It's like cutting half the workstations from a factory assembly line: the people upstream are still working, but the product never reaches quality control.

Circuit 2: GT (Greater-Than Comparison)

Task: Reasoning that requires comparing number magnitudes, e.g., "the war happened after 19XX."

Structure: year-encoding head (encoding year information in layers 9–11) → comparison head (performing comparison in layers 9–11).

Fate after distillation: Survived, and thriving.

• GPT-2 accuracy: baseline
• DistilGPT-2 accuracy: 87.7% (task performance recovery)
• Key mechanism: the teacher GPT-2's layer-9 comparison head was actively remapped to DistilGPT-2's layer 4
• Pearson correlation of activation patterns between the two: r = 0.945

This is not "residue" — it is "relocation." The student's layer 4 learned to imitate the teacher's layer 9, like a company moving from a 12-floor building to a 6-floor one: the finance department that used to occupy floor 9 no longer fits, so it is split and reassembled on floor 4. The new department has fewer staff, but its workflow is nearly identical to before.

Why Did One Die and the Other Live?

The answer lies in the paper's core insight:

> A circuit's compressibility determines its fate under distillation.

The IOI circuit is a depth-dependent pipeline — name-mover → S-inhibition → induction, with each link doing a specific job at a specific layer. Halving the layers cuts out the middle of the pipeline, and there is no way around it.

The GT circuit is depth-adaptable — its critical computation (the comparison head) was originally at layer 9, but the computation itself does not depend on "being at layer 9." It only needs "a set of attention heads performing comparison at some sufficiently deep layer." The 6-layer student found enough room at layer 4 to rebuild this computation.

In the paper's words:

> Distillation preserves circuits whose critical computations fit within the student's available depth, and destroys circuits whose pipelines extend beyond the student's depth.

What the Six Microscopes Saw

The strength of this paper is that it doesn't draw conclusions from a single method — it cross-validates with six different probes:

1. Linear CKA (representational similarity): Compares representational similarity layer by layer. DistilGPT-2's first 4 layers are highly similar to GPT-2's first 4 layers, but the last 2 layers diverge — it does the later work in a different way.

2. Logit lens: Projects hidden states to vocabulary space at each layer to see what the model is "thinking." The IOI circuit fails to form stable intermediate predictions in DistilGPT-2.

3. Attention pattern analysis: DistilGPT-2's attention heads are moderately correlated with GPT-2's, but "looking alike" does not mean "working alike" — IOI heads retain their patterns but lose their function.

4. SAE feature statistics: Sparse autoencoder features are largely shared between the two models, but their usage frequencies and combinations differ.

5. Activation patching: The most critical causal probe — transplanting GPT-2 activations into the corresponding DistilGPT-2 layers to see whether function is restored. IOI name-mover transplants remain ineffective because the downstream receivers no longer exist; GT comparison-head transplants into the student's layer 4 partially restore function.

6. Direct layer attribution (DLA): The GT circuit's contribution peak in DistilGPT-2 cleanly moves from the teacher's layer 9 to the student's layer 4 — direct evidence of "relocation."

What This Means

1. Aggregate metrics are "insufficient diagnostics"

The paper's sharpest conclusion: perplexity and downstream task accuracy are not enough to judge whether a distilled model has truly preserved its teacher's capabilities.

DistilGPT-2's IOI accuracy falls from 94.8% to 35.2% — a catastrophic drop. But if you only look at perplexity, MMLU, or other outcome-oriented benchmarks, you will never see this collapse. The model looks "about the same" while a critical circuit is already dead.

This echoes the "benchmark compliance blind spot" found in the Concordat paper: outcome-oriented evaluation systematically rewards "rule-breaking but correct outcomes" while ignoring "correct outcomes with collapsed processes."

2. Distillation is not uniform compression — it is selective destruction

We tend to imagine distillation as shrinking a high-resolution photo into a lower-resolution one — information loss is uniform, but overall structure is preserved. This paper shows distillation is more like an earthquake: some buildings (circuits) are compact enough to merely shift position; others depend on specific foundation depth and collapse outright.

3. Circuit compressibility is a new dimension

The paper introduces a new concept: circuit compressibility. Whether a circuit can be compressed depends not on how important it is, but on whether its computational pipeline can be reorganized within fewer layers.

This points to a new direction for model compression: not "how much did perplexity degrade after compression," but "which circuits will collapse after compression." If you know a key capability depends on a deep pipeline like IOI, you know you cannot simply halve the layer count.

Honest Boundaries

The paper does not claim to solve everything. The authors candidly list several limitations:

• Only two circuits (IOI and GT) were studied; the compression behavior of more circuits remains to be verified
• The GT circuit's "survival" verification is incomplete — while DLA and activation patching both support the "relocation" hypothesis, full causal intervention experiments have not yet been done
• The SAE reference bases are not fully comparable across the two models, so feature comparisons should be interpreted cautiously

This honesty makes the paper stronger. A paper claiming to "completely solve distillation" would be suspect; a paper saying "we used six methods to examine two circuits, found a dichotomy, but much remains unexamined" is solid science.

One-Sentence Summary

Distillation is not uniform compression but selective destruction: circuits that can relocate will relocate (GT: 87.7% recovery), and circuits that cannot will die (IOI: 35.2% collapse). And outcome-only benchmarks will never see the earthquake.

---

Paper link: arXiv:2610.37157

Related tools:

• TransformerLens — mechanistic interpretability framework
• SAELens — sparse autoencoder analysis tool