SubFit: Moving Beyond Layer Removal to Submodule-Level LLM Compression

Forum topic · 小凯 · 2026-06-02

Summary

SubFit is a new LLM compression method that challenges the dominant layer-removal paradigm. Instead of deleting entire Transformer layers, SubFit operates at the submodule granularity, independently selecting and replacing redundant Attention and FFN components with non-contiguous selection. The authors show that redundancy in pretrained Transformers is unevenly distributed: Attention and FFN redundancy can differ sharply within the same layer, and redundant submodules are often scattered across non-adjacent layers. SubFit approximates removed Attention submodules with low-rank bypasses, replaces FFN submodules with a shared low-rank basis, and fits replacement parameters using calibration data without full retraining. Across 10 LLMs and sparsity levels from 12.5% to 37.5%, SubFit at 25% sparsity retains 84.6% downstream accuracy (vs 81.6% for the strongest baseline) with 2.42x perplexity degradation vs 4.34x. At 37.5% sparsity it reduces perplexity degradation by 5.69x over baselines and is the only method preserving over 73% accuracy on instruction-tuned models. It also delivers measurable inference speedups and KV-Cache savings.

If you were pruning a tree, how would you do it?

The most intuitive approach: saw off entire branches, from top to bottom. Crude, but effective — at least for trees.

But when "slimming down" a large language model, the same strategy isn't necessarily optimal. For years, the mainstream approach in LLM compression has been whole-layer removal — find the least important layers, remove them entirely, then use compensation mechanisms to make up the loss. Like pruning fruit trees, one branch at a time.

But the SubFit paper poses a counterintuitive question: why must we delete whole layers? Is a model's redundancy really distributed by layers?

The answer is no. And based on that negative answer, SubFit pushes LLM compression granularity from "layer" down to "submodule" — Attention and FFN blocks can be independently and non-contiguously selected and replaced.

Two Constraints in Existing Methods

Before SubFit, all "replacement-based" compression methods shared two implicit assumptions:

Constraint 1: Whole-layer granularity. The basic unit of compression is an entire Transformer layer. Keep it or replace it — no middle option.

Constraint 2: Contiguous selection. Removed layers must be contiguous. Deleting layers 8, 9, 10 is fine; deleting layers 3, 7, and 12 is not.

Where do these constraints come from? Mostly engineering convenience — whole-layer replacement is simple to implement, and contiguous selection makes residual-based compensation easier. But the SubFit authors found these constraints don't hold up theoretically.

Redundancy Isn't Distributed by Layers

By analyzing the redundancy of individual submodules in pretrained Transformers, the authors identified two key facts:

Fact 1: Within the same layer, Attention and FFN redundancy differ dramatically. Some layers have critical Attention but redundant FFNs; in others it's reversed. Whole-layer deletion forces you to handle two components with completely different redundancy at once — either you delete too little (wasting the FFN's compression headroom) or too much (damaging critical Attention capabilities).

Fact 2: Redundant submodules aren't necessarily contiguous. The FFN in layer 3 and the FFN in layer 17 might both be redundant, while layers in between are important. Forcing contiguous selection means either abandoning these scattered redundancies or removing important layers along with them.

It's like cleaning a room: the garbage isn't concentrated in one corner — it's scattered everywhere. Cleaning only one area leaves trash elsewhere; cleaning thoroughly means checking every corner.

SubFit: Precision Surgery at the Submodule Level

SubFit's core idea: break both the whole-layer and contiguity constraints, and perform precise compression at the submodule level.

Specifically:

1. Non-contiguous selection. Attention and FFN redundancy are assessed independently, selecting the least important submodules of each. The FFN in layer 3 and the FFN in layer 17 can both be selected, free of contiguity constraints.

2. Differentiated replacement. Attention and FFN use different replacement strategies:

Attention submodules can be approximated with a simple low-rank bypass, because Attention output is essentially a weighted sum of Value vectors, with information concentrated in a low-rank subspace.
FFN submodules need higher-rank mappings, but multiple removed FFNs can share a single low-rank basis, controlling deployment cost.

3. Calibration-data fitting. Replacement module parameters are fitted with a small amount of calibration data — no full retraining needed.

An analogy: if whole-layer removal is smashing walls with a sledgehammer, SubFit is a scalpel precisely excising diseased tissue while stitching an appropriate patch on each incision — a thinner patch for Attention, a thicker one for FFN, but multiple FFNs share the same patch cloth to save material.

The More Aggressive the Compression, the Bigger the Advantage

SubFit was comprehensively evaluated on 10 LLMs (5 base models + 5 instruction-tuned models) at 5 sparsity levels (12.5% to 37.5%).

Key numbers:

At 25% sparsity: SubFit retains 84.6% of downstream task accuracy, with perplexity degrading only 2.42x. The strongest baseline retains only 81.6% accuracy, with 4.34x perplexity degradation.
At 37.5% sparsity: the gap widens further. SubFit's perplexity degradation is 5.69x lower than the strongest baseline, and it's the only method preserving over 73% accuracy on instruction-tuned models.
At 12.5% sparsity: the gap is smallest (0.11x), because whole-layer removal doesn't hurt much under light compression anyway.

This trend makes sense: the more aggressive the compression, the more obvious the advantage of precision surgery. Like losing weight — dropping 5 pounds works any way you try, but dropping 30 pounds, the difference between even fat loss and randomly cutting muscle is huge.

Practical Gains: Not Just Accuracy

SubFit also delivers real inference speedup and KV-Cache savings. Because it removes specific submodules rather than whole layers:

Every skipped Attention layer saves one attention computation at inference
Every skipped FFN saves two matrix multiplications
KV-Cache only needs to store key-value pairs for retained Attention layers

At 25% sparsity, SubFit achieves measurable inference speedup along with significantly reduced KV-Cache footprint.

The Deeper Insight: An "Anatomy" of AI Models

SubFit's significance isn't just an incremental compression improvement. It reveals something deeper: the redundancy structure of Transformers is more complex and uneven than we assumed.

Whole-layer removal is popular not because it's optimal, but because it's simplest. Like bloodletting in early medicine — not because it worked, but because understanding of the circulatory system was insufficient.

What SubFit does amounts to a fine-grained "anatomy" of Transformers — discovering that Attention and FFN redundancy patterns are entirely different, and that redundancy distribution is uneven across depths. Only with this finer understanding can more precise compression strategies be designed.

The idea extends further: if submodule-level compression beats whole-layer removal, what about token-level, attention-head-level compression? Our understanding of Transformer internals may still be far from complete.

---

Paper: From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression Code: github.com/eliacunegatti/SubFit Authors: Elia Cunegatti, Marcus Vukojevic, Erik Nielsen, et al.