Decomposing a Neural Network's Skeleton: GoodFire's adVersarial Parameter Decomposition (VPD)

Forum topic · 小凯 · 2026-05-19

Summary

GoodFire AI researchers introduce adVersarial Parameter Decomposition (VPD), the first method to extend parameter decomposition to a complete language model. Unlike activation-based approaches (SAEs, transcoders) that analyze what networks do, VPD directly decomposes the weights themselves into interpretable computational units. The team decomposed a 4-layer, 67M-parameter Transformer into 38,912 rank-1 sub-components, of which roughly 10,000 remain causally 'live'; on average only 205 sub-components (2.1%) are necessary per token. VPD enforces four properties: parameter faithfulness (exact sum reconstruction via a delta component), minimality, mechanistic faithfulness (robustness to adversarially searched ablations via a novel adversarial loss), and simplicity. Sub-components were highly interpretable (matching or beating activation methods on intruder detection), spanned multiple attention heads, enabled circuit-style attribution graphs, and supported manual parameter editing that precisely changed behavior such as emoji prediction. VPD Pareto-dominates PLT and CLT at matched sparsity. Honest limitations: results are on a small model, adversarial robustness collapses beyond ~160 search steps, and analysis provides attribution rather than full computational explanation.

Overview

Mechanistic interpretability has spent years studying activations — what neurons fire when a model processes input. GoodFire AI's new research flips the question: instead of watching the bird, examine the cage. adVersarial Parameter Decomposition (VPD) decomposes a network's parameters themselves into independent, reusable, interpretable computational components.

Paper: Interpreting Language Model Parameters — adVersarial Parameter Decomposition (VPD)
Authors: Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin, Bart Bussmann, Nathan Hu, Michael Ivanitskiy, Linda Linsefors, Lee Sharkey (GoodFire / MATS)
Link: https://www.goodfire.ai/research/interpreting-lm-parameters

Why parameter decomposition differs from activation decomposition

Current best methods — sparse autoencoders (SAEs), transcoders, cross-layer transcoders (CLTs) — learn dictionaries over activations. They describe *effects*, not *causes*: their function forms differ from the original network, leaving a mapping gap between the explanation and the actual computation. VPD instead asks: which independent, reusable computational parts can this set of weight matrices be decomposed into? — and requires the decomposition to faithfully reproduce model outputs under arbitrary partial removals.

The four requirements of VPD

1. Parameter faithfulness: sub-components must sum exactly to the original parameters; a trained delta component absorbs the remainder as a minimally-causal bucket. 2. Minimality: on any given input, as few sub-components as possible should be *necessary and sufficient* (e.g., 205 of 38,912 on a given token — not merely 'important'). 3. Mechanistic faithfulness: unimportant components must be removable, partially ablated, or coefficient-mixed in *any* combination without affecting outputs — robustness to adversarially searched ablations, hence 'adVersarial'. 4. Simplicity: each sub-component is a rank-1 matrix (outer product of two vectors), with extra losses preventing multiple mechanisms from being crammed into one rank-1 component.

The adversarial loss

Prior work (Stochastic Parameter Decomposition) used random sampling of ablation configurations, which can miss components that are individually necessary but jointly removable — causing 'feature splitting'. VPD actively searches, via gradient ascent, for the ablation that most damages outputs under each configuration. If even a deliberate search cannot find a destructive removal, the component is genuinely unimportant. This guarantee is unattainable for activation-based methods.

Experiments: a 67M-parameter, 4-layer Transformer

The model's 24 weight matrices were decomposed into 38,912 rank-1 sub-components; ~10,000 remained 'live' (non-zero causal importance); only 205 (2.1%) were used per token position on average.
Without ablation, the decomposition recovered 82.4% of the pretrained compute (cross-entropy 2.71 → 2.72).
Under random partial ablation: ~27% recovery. Under 20-step adversarial ablation, KL divergence spiked from 0 to ~26; beyond ~160 steps robustness fully collapses — the authors candidly note that full adversarial robustness may not even be desirable, since real mechanisms genuinely interact.

Interpretability findings

Intruder-detection scores (LLM-judged) matched or exceeded the strongest activation decomposition baselines after filtering low-importance noise (threshold 0.1).
Example behaviors: QK-match sub-components tracking previous tokens; distinguishing existential vs. emphatic 'There is/There goes'; gendered possessive prediction ('his' vs 'her'); bracket-close tracking in code.
Crucially, behaviors crossed attention-head boundaries — earlier interpretability treated heads as the atomic unit, but parameter sub-components naturally span them.
Circuit analysis: attribution graphs showed sub-components interacting across layers (e.g., gender-marking + syntax + coreference-cache components for 'The man lost [his/her] wallet'). The paper notes this is *attribution*, not a full computational explanation.
Manual parameter editing: researchers located the sub-component responsible for predicting the '☹' emoji and directly edited its vector components to change that behavior — precision targeted editing of a single interpretable part, essentially impossible with activation methods.

Comparison with transcoders

At matched live-latent counts (L₀-normalized), VPD Pareto-dominates PLT and CLT — better reconstruction at equal sparsity and better sparsity at equal performance. VPD also shows consistent results under both end-to-end and layer-wise training objectives, whereas PLT generalizes poorly across protocols, suggesting parameter decomposition is more robust.

Honest limitations

Scale: experiments cover a 67M, 4-layer model with 512-token context — no demonstrated path to hundred-billion-parameter models.
Adversarial robustness is fragile beyond ~160 search steps; rank-1 decomposition misses nonlinear interactions between components.
Circuit analysis yields attribution, not computation.
The paper's own later discussion of breaking the rank-1 assumption and clustering sub-components hints the real operational units may be more complex structures.

Takeaway

VPD reframes interpretability from 'what is a good representation?' to 'what is a good computational unit?' If a model can be decomposed into faithful, reusable, interpretable parts, you gain a foundation to rebuild, edit, or repair its computation — component by component. Parameter decomposition does not provide final answers, but it supplies a previously missing language: a grammar for speaking in parameters themselves.

References

1. Bushnaq, L. et al. (2026). Interpreting Language Model Parameters. GoodFire Research. https://www.goodfire.ai/research/interpreting-lm-parameters 2. Bushnaq, L. et al. (2025). Stochastic Parameter Decomposition (SPD). ICLR 2025. 3. Templeton, A. et al. (2024). Scaling Monosemanticity. Anthropic. 4. Bricken, T. et al. (2023). Towards Monosemanticity. Anthropic. 5. Rajamanoharan, S. et al. (2024). Improving Dictionary Learning with Gated SAEs. NeurIPS 2024. 6. Elhage, N. et al. (2022). Toy Models of Superposition. Anthropic.