SubFit: From Layer Dropping to Submodule-Level Surgery in LLM Compression

Forum topic · 小凯 · 2026-06-02

Summary

SubFit is a new LLM compression method that abandons two hidden assumptions of replacement-based compression: whole-layer granularity and contiguous layer selection. By analyzing redundancy across Transformer submodules, the authors show that Attention and FFN blocks within the same layer differ greatly in redundancy, and that redundant submodules are often non-contiguous. SubFit therefore selects and replaces Attention and FFN submodules independently: Attention blocks are approximated with a low-rank bypass, while multiple removed FFNs share a common low-rank basis fitted on a small calibration set. Evaluated on 10 LLMs (5 base, 5 instruction-tuned) at sparsity levels from 12.5% to 37.5%, SubFit retains 84.6% downstream accuracy at 25% sparsity (vs 81.6% for the strongest baseline) with only 2.42x perplexity degradation versus 4.34x. At 37.5% sparsity the gap widens to 5.69x, and SubFit is the only method keeping over 73% accuracy on instruction-tuned models. It also delivers measurable inference speedups and KV-Cache savings.

If you were pruning a tree, the intuitive approach would be to saw off entire branches from top to bottom—crude but effective. For years, mainstream LLM compression has followed the same strategy: delete whole layers deemed least important, then compensate for the loss. But what if redundancy in a model doesn't align neatly with layer boundaries?

SubFit (arXiv:2606.02559, code) asks exactly this counterintuitive question and answers it: no. Its contribution is shifting compression granularity from *layers* to *submodules*—Attention and FFN blocks can be selected and replaced independently and non-contiguously.

Two Hidden Constraints in Prior Work

All prior replacement-based compression methods share two assumptions:

1. Whole-layer granularity: the unit of compression is an entire Transformer layer—keep it or replace it. 2. Contiguous selection: removed layers must be consecutive (e.g., layers 8–10, not 3, 7, and 12).

These constraints exist mainly for engineering convenience: whole-layer replacement is simple, and contiguous selection makes residual-based compensation easy. But SubFit's authors find both theoretically unjustified.

Redundancy Is Not Distributed by Layer

Analyzing submodule redundancy in pretrained Transformers reveals two key facts:

Within a layer, Attention and FFN redundancy differs dramatically. Some layers have critical Attention but redundant FFNs, and vice versa. Deleting whole layers forces you to treat both identically—either under-compressing or damaging important capacity.
Redundant submodules are often non-contiguous. The FFNs at layers 3 and 17 may both be redundant while the layers between them are important. Forced contiguity means either abandoning scattered redundancy or removing important layers too.

Like cleaning a room where the trash is scattered everywhere, not piled in one corner.

SubFit: Precision Surgery at the Submodule Level

SubFit breaks both constraints:

1. Non-contiguous selection: Attention and FFN blocks are independently scored for redundancy; e.g., the FFNs of layers 3 and 17 can both be removed. 2. Differentiated replacement:

Attention blocks need only a low-rank bypass, since Attention output is a weighted sum of Value vectors concentrated in a low-rank subspace.
FFN blocks require higher-rank mappings, but multiple removed FFNs share a single low-rank basis to control deployment cost.

Calibration-data fitting

Where whole-layer deletion uses a sledgehammer, SubFit uses a scalpel—with thinner patches for Attention, thicker ones for FFN, and shared patch material across FFNs.

The More Aggressive the Compression, the Bigger the Gain

SubFit was evaluated on 10 LLMs (5 base + 5 instruction-tuned) across 5 sparsity levels (12.5%–37.5%):

At 25% sparsity: SubFit retains 84.6% of downstream task accuracy with perplexity degradation of only 2.42x. The strongest baseline retains 81.6% accuracy with 4.34x degradation.
At 37.5% sparsity: the gap widens—SubFit's perplexity degradation is 5.69x lower than the strongest baseline, and it is the only method retaining over 73% accuracy on instruction-tuned models.
At 12.5% sparsity: differences are minimal (0.11x), since light compression costs little for any method.

Practical Benefits Beyond Accuracy

Because SubFit removes specific submodules rather than layers:

Each skipped Attention block saves one attention computation per token
Each skipped FFN saves two matrix multiplications
KV-Cache only needs to store key-value pairs for retained Attention layers

At 25% sparsity, SubFit achieves measurable inference speedup and significantly lower KV-Cache footprint.

The Bigger Picture: An "Anatomy" of AI Models

SubFit's deeper significance is that it reveals Transformer redundancy is more complex and uneven than assumed. Whole-layer deletion is popular not because it's optimal, but because it's simple. By dissecting the model at submodule granularity—showing Attention and FFN redundancy patterns differ completely, and that distributions vary by depth—SubFit enables genuinely targeted compression. And the question extends further: what about token-level or attention-head-level granularity? Our understanding of Transformer internals may still be far from complete.

---

Paper: From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression Code: github.com/eliacunegatti/SubFit Authors: Elia Cunegatti, Marcus Vukojevic, Erik Nielsen, et al.