Ghosted Layers: Training-Free Closed-Form Activation Alignment for Recovering Layer-Pruned LLMs

Forum topic · 小凯 · 2026-05-19

Summary

Layer pruning—directly removing entire Transformer blocks from a large language model—is one of the most aggressive compression strategies, but it breaks the input distribution that downstream layers were trained on. A new approach called 'Ghosted Layers' by Yun, Jo, Karimireddy, and Lee (arXiv:2605.15491) repairs this damage without any training. Instead of inserting a small learned neural network at the pruned position—which is limited to a hand-picked operator family—the method uses a few hundred calibration samples to solve a closed-form, globally optimal linear operator that reconstructs the activation shift introduced by removing the layer. Because the solution is unconstrained over the full linear space, it avoids the suboptimality of restricted parameter subspaces. Experiments across multiple LLM backbones and pruning strategies show consistent improvements in accuracy and perplexity over prior training-free baselines, with zero additional inference cost. Open questions include how calibration set size affects recovery quality and whether a linear surrogate suffices for deeply specialized late layers, given the inherent limits of approximating nonlinear Transformer blocks.

Layer pruning—directly deleting a few Transformer blocks—is the most straightforward form of LLM compression, and also the most destructive. Once a layer is removed, the input distribution of the next layer no longer matches what it saw during training. It is like pulling out several rungs from the middle of a ladder: whoever stands on top cannot make the step.

Prior approaches and their limits

The standard fix inserts a small neural network at the pruned position to compensate for the missing block. But this is constrained to a parameter subspace: the hand-selected operator family may not even lie in the neighborhood of the optimal solution.

The Ghosted Layers approach

The method of Yun, Jo, Karimireddy, and Lee requires no training at all. Using a small calibration set (a few hundred samples), it solves for a closed-form optimal linear operator that reconstructs the activation discrepancy introduced by the pruned layer. Key properties:

The solution is the global optimum of an unconstrained alignment objective—no restriction to a specific operator family; the best transformation is solved directly over the entire linear space.
Experiments across multiple LLM backbones and pruning strategies show consistent improvements over prior training-free baselines, in both accuracy and perplexity.
The efficiency benefit of layer pruning is preserved: no additional inference cost.

Open questions

How does calibration set size affect recovery quality? The paper uses "a few hundred samples"—exactly how many are needed?
Where is the limit of approximating a nonlinear Transformer layer with a linear operator? If the pruned layer is deep and highly functionally specialized, is linear compensation enough?

References

1. Yun, V., Jo, J., Karimireddy, S. P., & Lee, S. (2026). *Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs*. arXiv:2605.15491 [cs.LG]. 2. Menick, J., et al. (2024). *The Capacity for Moral Self-Correction in Large Language Models*. arXiv. 3. Frantar, E., & Alistarh, D. (2023). *SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot*. ICML.