A Is for Absorption: When Interpretability Tools Distort What They Measure
> Feynman's insight cuts both ways: if you cannot explain something, you do not understand it. But this post covers the reverse problem — the tool we use to *understand* LLM internals has major defects of its own.
Background: SAEs and feature splitting
Large language models encode information in high-dimensional vectors that mix many concepts at once. Sparse Autoencoders (SAEs) decompose these vectors into sparse, human-interpretable feature directions — e.g., feature 314 = "algebra," feature 271 = "geometry," feature 42 = the broader concept "mathematics."
The mainstream assumption in mechanistic interpretability: give the SAE a larger capacity budget and coarse features automatically split into finer ones:
- Small SAE: "mathematics" (one feature)
- Medium SAE: "algebra," "geometry," "calculus"
- Large SAE: "linear algebra," "abstract algebra," "Euclidean geometry," "differential geometry," ...
- Title: A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
- Authors: David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, Joseph Isaac Bloom
- Venue: NeurIPS 2025 (Oral)
- Link: OpenReview
- Keywords: SAE, interpretability, feature splitting, LLM
This seems to promise unlimited refinement just by adding compute. A NeurIPS 2025 Oral paper shows the picture is broken.
The absorption problem
Train a bigger SAE and the parent feature "mathematics" still exists — but it stops firing on math content. Feed in an algebra problem: only the child feature "algebra" activates; the parent stays silent, even though algebra is math.
It is like a family tree where the "Zhang family" node stops recognizing Zhang San and Zhang Si once the tree is drawn in more detail.
Why it happens: sparsity
SAE training maximizes sparsity — few features active at once. If "algebra" fires, also firing "mathematics" wastes the sparsity budget. The optimizer chooses the cheaper option: silence the parent. The loss function cares about sparsity, not semantic consistency.
Worse, absorption is irreversible: once a parent feature is hollowed out, changing SAE size or sparsity cannot restore it. The paper validates this across hundreds of LLM SAEs.
Why it matters
1. Unreliable feature maps — inspecting which features fire may miss absorbed parents; you conclude the model "isn't thinking about math" when it is. 2. Unreliable causal interventions — steering the model via an absorbed parent feature fails, because it no longer does any work. 3. Questioned scaling promise — "bigger SAEs give better decompositions" may just produce more absorbed features.
Possible directions
The paper offers no ready-made fix, but suggests:
1. Better training objectives — add hierarchy-consistency constraints to the sparsity + reconstruction loss. 2. Post-hoc detection — a metric to scan for absorbed features before deploying an SAE. 3. Structural priors — protect known hierarchies ("algebra" ⊆ "mathematics") during training. 4. Relax pure sparsity — the sparsest decomposition may not be the most interpretable one.
Takeaway
SAEs are not a passive microscope — they are an active filter. Before trusting interpretability results, the field must ensure the interpretation tools themselves are not systematically distorting the structure they claim to reveal.
---
Paper info