| Item | Detail | |------|--------| | Title | Interpreting Language Model Parameters — adVersarial Parameter Decomposition (VPD) | | Authors | Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin, Bart Bussmann, Nathan Hu, Michael Ivanitskiy, Linda Linsefors, Lee Sharkey (GoodFire / MATS) | | Link | https://www.goodfire.ai/research/interpreting-lm-parameters | | Date | May 5, 2026 | | Core contribution | First parameter decomposition method extended to a full language model: a 4-layer 67M-parameter Transformer decomposed into 38,912 interpretable subcomponents; introduces an adversarial reconstruction loss; subcomponents are free of feature splitting, highly interpretable, and support circuit analysis and manual parameter editing | | arXiv / GitHub | See references at the end |
Imagine a birdcage in a park. You stand outside watching the birds fly inside it — you can observe different birds behaving differently — but you can't see the cage's structure. You know there are bars, hinges, and latches, but your attention is on the birds, not the structure.
That's the situation mechanistic interpretability has been in for the past few years.
The field has focused on neural network activations — what pulses each layer's neurons produce when the model receives an input. It's like watching the birds' behavior — seeing which neurons "light up," which attention heads "attend" — but never examining the cage itself: what the model's parameters actually look like, how they're structured, and how they compose into algorithmic parts.
GoodFire AI's latest research offers an answer — perhaps attacking the problem from a fundamentally different direction. The method is called adVersarial Parameter Decomposition (VPD).
Instead of observing the network's "behavior" (activations), it directly decomposes the network's "skeleton" (the parameters themselves).
If you can't decompose the cage's skeleton, you'll never know why the birds fly the way they do.
Activation Decomposition vs. Parameter Decomposition — Fundamentally Different Assumptions
To understand why VPD matters, first consider what today's popular "activation decomposition" methods do.
In a neural network, information flows between layers. You feed in text → some neurons activate → output vectors pass to the next layer → more complex neurons activate → a prediction emerges. This is the "flow of activations."
Today's best decomposition methods — transcoders, sparse autoencoders (SAEs), cross-layer transcoders (CLTs) — look for patterns in this flow. They train a "dictionary" on a layer's activations, decomposing complex activation vectors into a few basic, reusable basis vectors. It's like a compression format describing what the model does for each input.
The problem: these dictionaries describe "effects" — what happened to activations — not "causes" — how the parameters compute them. Transcoders use a functional form different from the original network's. This creates a "mapping gap" between their explanations and the actual machine: you can explain a simplified proxy, but can't precisely map back to the original parameters' computation.
VPD takes a completely different path. Rather than starting from activations, it starts from parameters, directly asking: which independent, reusable computational parts can this set of weight matrices be decomposed into? And it requires this decomposition to faithfully reproduce the model's output even when components are partially removed in various ways.
The Math — Four Hard Requirements
VPD's training objective requires four properties simultaneously. If any one fails, the decomposition is unreliable.
Parameter Faithfulness: All decomposed subcomponents must sum together to exactly reconstruct the original parameters. Not approximately — exactly. The paper uses a "Δ component" to absorb any residual difference — "a bucket for parts we don't understand" — trained to be as small and causally insignificant as possible.
Minimality: On any given input, the fewer subcomponents involved in computation, the better. In principle you could skip decomposition — "the whole model is one big component" — but that's useless. A good decomposition means: on a given sentence, only 205 subcomponents are necessary (2.1% of the total), and the rest can be partially or fully removed without changing the output. Those 205 aren't just "important" — they are "necessary and sufficient."
Mechanistic Faithfulness: A sharper requirement. Not just "removing unimportant components leaves the output unchanged" — but "any combination of unimportant components — partial removals, coefficient mixtures — should have no effect in any subset." Components should be immune to arbitrary damage if and only if they are truly unimportant. The mathematical name for this property is "adversarial robustness to ablation" — the source of the method's name.
Simplicity: Each subcomponent should be as simple as possible. Each is a rank-1 matrix — an outer product of two vectors — the simplest possible matrix decomposition. But if rank-1 isn't enough, extra loss terms prevent multiple independent mechanisms from being "squeezed" into one rank-1 component (since the sum of two rank-1 matrices can still be rank-1).
Adversarial Loss — The Core Technical Innovation
Earlier we required that removing unimportant components leaves the output unchanged. But under how many removal configurations should this hold?
Prior work (Stochastic Parameter Decomposition, SPD) used random sampling — randomly deciding how much to remove of each unimportant component. The paper argues this isn't good enough: if two components can be "removed together but not individually," random sampling may occasionally hit the joint-removal case without ever testing individual removal — wrongly marking both components as unimportant, producing "feature splitting."
VPD's key innovation is an adversarial loss: under each removal configuration, gradient ascent actively searches to maximize output damage. Not "let's try and see" — but "we deliberately search for the most destructive removal; if even this search can't break the output, the component really is unimportant."
This is an extremely strong guarantee of mechanistic faithfulness — one that activation-based methods fundamentally cannot provide, because they are not parameter-centric.
Experiments: Decomposing a 67M-Parameter, 4-Layer Transformer
The paper targets a small but complete language model — 4 layers, 67M parameters, 768-dim embeddings — trained on a copyright-free Pile subset.
The model's 24 weight matrices (per-layer Q, K, V, O attention matrices plus MLP up/down matrices) were decomposed into 38,912 rank-1 subcomponents. At the end of training, only about 10,000 are "alive" (non-zero average causal importance). The average token position uses only 205 of them (2.1%).
Key numbers:
- Under no-ablation conditions, the decomposed model recovers 82.4% of the target model's pretrained compute (cross-entropy 2.71 → 2.72)
- Under random partial ablations, it recovers about 27% of pretrained compute
- Under 20 steps of adversarial ablation, KL divergence jumps from 0 to ~26
- Beyond 160 adversarial steps, robustness cannot be maintained
- Some attention subcomponents specifically monitor "the previous token" — detecting QK matches
- Some distinguish "existential vs. deictic constructions" ("There is a cat" vs "There goes my cat") — a sophisticated grammatical distinction
- Some are active when predicting gendered possessive pronouns ("his" vs "her")
- Some track bracket closure
- Gendered possessive pronoun prediction ("The man lost [his/her] wallet"): the circuit involves multiple cross-layer subcomponents — one detecting gender markers, one handling grammatical structure, one caching coreference chains
- Bracket closure (in code data): one cross-head subcomponent tracks open brackets; another closes them at the right positions
Honest labeling: The paper candidly admits adversarial robustness fully collapses past ~160 steps. The independence assumption among subcomponents fails under extreme search — certain interdependencies exceed what rank-1 components can capture. The authors themselves say "full adversarial robustness isn't necessarily desirable" — real mechanisms genuinely interact.
Why the Subcomponents Are Highly Interpretable
The paper measures interpretability via an "intruder detection" test: given texts where a subcomponent is highly causally active, plus an "intruder" text where it isn't active, an LLM judge must find the intruder.
After filtering low-importance noise (threshold 0.1), VPD subcomponents score equal to or better than the strongest activation-decomposition methods. Crucially, these explanations correspond to real parameter structure, not a compressed representation of activations.
Concrete examples from interactive panels:
Most impressively, these behaviors span multiple attention heads. Interpretability research has long treated "attention heads" as the basic unit of analysis — but VPD shows parameter subcomponents naturally cross head boundaries, closer to how real neural systems work: functional units in prefrontal cortex aren't confined to one "head."
Circuit Analysis — Information Flow Between Subcomponents
The paper also builds "attribution graphs" — showing how subcomponents interact on specific inputs. This is a new form of circuit analysis: not "neuron connects to neuron," but "subcomponent interacts with another through attention and MLP matrices."
Two case studies:
Hand-Tuning Parameters — Something You'd Never Have Thought Possible
A striking demonstration: manually editing parameters to change model behavior.
Specifically: the training data contained emoji-like characters such as "☹", and the model learned to predict them in appropriate contexts. Researchers identified which subcomponent was responsible — then directly modified the vector constituents of that subcomponent, changing the emoji-prediction behavior.
This is not transfer learning. This is not fine-tuning. This is a human directly editing values inside a vector parameter — repairing a neural network like repairing a watch. This precision — locating a single interpretable computational part and editing it — is extremely difficult for any activation-based decomposition method.
Does VPD Beat Transcoders?
The paper runs careful comparisons. Core conclusion:
At matched numbers of active components/latents (L₀-normalized), VPD Pareto-dominates PLT and CLT — for equal compute, VPD restores performance more sparsely; at equal sparsity, VPD restores more performance.
Another key finding: VPD decompositions show better consistency between "end-to-end" and "layer-wise" training objectives. PLT performs best layer-wise but generalizes poorly end-to-end; VPD performs similarly under both. This suggests parameter decomposition is more robust — bound directly to parameter structure, less sensitive to changes in evaluation protocol.
Honest Limitations
First: the paper decomposes a 67M-parameter, 4-layer model — not GPT-4 or Claude Sonnet. Scaling VPD to hundred-billion-parameter models would require enormous compute; the paper doesn't establish basic feasibility of that path.
Second: adversarial robustness remains fragile. After ≥160 adversarial steps, outputs are significantly damaged — the independence assumption fails under long searches. Real nonlinear interactions between subcomponents are missed by the rank-1 decomposition.
Third: circuit analysis provides "attribution" graphs, not computational graphs. The paper explicitly distinguishes the two — attribution tells you which subcomponents matter, not what computation they perform. VPD makes finding "important parts" easier, but still doesn't directly explain how they compute.
Fourth: the model uses a 512-token context, far smaller than modern deployed models, possibly limiting the generality of the findings.
My Take
This paper doesn't just propose a new method. It surfaces a neglected question:
The past five years of mechanistic interpretability have focused almost entirely on activation space — transcoders, SAEs, linear probes — all studying the "shape" of activations. But those shapes are produced by parameters. Like staring at ripples on water without ever diving to see the terrain that generates them.
VPD's value is reframing the question from "what is a good representation" to "what is a good computational unit." Representations are byproducts — computation is the engine. If you understand how computational units are built and interact, you can manually edit, repair, and redesign them.
And — once you can decompose a model into a set of reusable, interpretable, faithful computational parts — you have a launchpad: you can rebuild the model's computation from here, one part at a time. Not fine-tuning — reinvention.
The most important ideas may be buried in the paper's later sections — on "when to break the rank-1 assumption" and "subcomponents clustering into components" — because the real operational units of neural networks may not be rank-1 but something more complex. Rank-1 is an approximation made for simplicity — but what if the true mechanisms are rank-2?
Parameter decomposition won't give you the final answer — but it gives you a language that didn't previously exist: a grammar for speaking in parameters themselves.
References
1. Bushnaq, L. et al. (2026). Interpreting Language Model Parameters. GoodFire Research. https://www.goodfire.ai/research/interpreting-lm-parameters 2. Bushnaq, L. et al. (2025). Stochastic Parameter Decomposition (SPD). ICLR 2025. 3. Templeton, A. et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Anthropic. 4. Bricken, T. et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Anthropic. 5. Rajamanoharan, S. et al. (2024). Improving Dictionary Learning with Gated Sparse Autoencoders. NeurIPS 2024. 6. Elhage, N. et al. (2022). Toy Models of Superposition. Anthropic.