Learned Features, Half of Them Neurons: Auditing Transcoder Feature Dictionaries

The easiest entry point for explaining a neural network is to look at its internal "features." But direct inspection fails: a single neuron fires in completely different contexts — "Paris," "capital," "Eiffel Tower" — a phenomenon researchers call polysemanticity. One neuron, many meanings, no readable conclusion.

So a few years ago the interpretability community switched tools: instead of using neurons as the unit of analysis, they trained sparse replacement models to imitate the original MLP layer, with each replacement unit ideally activating in only one context. These replacement models are called transcoders — public feature dictionaries for large models (the Gemma Scope series) were built this way. Circuit analysis, attribution graphs: mainstream methods from the past two to three years all rest on these "learned features."

A paper posted to arXiv on October 17, 2026 (arXiv 2610.24521, Stepan Shabalin and Mateusz Piotrowski, Anthropic Fellows with Georgia Tech and Anthropic) gives this default approach a health check. The conclusion stings a bit: the features doing the heavy lifting in these dictionaries are mostly neurons with new names.

Where It Started: Two Prior Studies Exposed the Problem

The story begins with two earlier works. Arora et al. (2026) found that attribution done directly on raw MLP neurons — without learned features — could recover circuits as sparse and faithful as feature-dictionary circuits in some settings. Ye et al. (2025) went further: treating thresholded activations of raw neurons as a "dictionary" matched learned dictionaries on standard benchmarks.

If bare neurons perform this well, are transcoders a necessary tool or an expensive ritual? The paper's approach: put both under the same ruler and see what features actually add over neurons.

What "A Feature Is a Neuron" Means

First, an operational criterion. The paper computes each feature's encoder and decoder directions and their cosine similarity with the best-matching MLP neuron; a feature counts only if both directions exceed 0.7 against the same neuron — and this is not just weight similarity: activation patterns and circuit roles must match too. On per-feature scatter plots of most transcoders, features meeting this criterion cluster separately from the rest, visibly distinct.

The paper names these neuron features.

The Numbers: A Minority of the Dictionary, a Majority of Activations

The results are clearest as a set of numbers. The paper scanned 13 open-source transcoders across the Gemma, Qwen, and Llama families. For Gemma Scope 2's small dictionary (262k features) on Gemma 3 4B: neuron features are only 8.9% of the dictionary but carry 63.8% of activations. A few carrying most of the load.

Scaling from 1B to 27B parameters, this contrast grows monotonically: dictionary share 5.9% → 8.9% → 15.6% → 37.1%; activation share 57.9% → 63.8% → 82.8%. Larger models mean more neuron features carrying activations — one mundane reason: with a fixed dictionary width, more neurons means fewer features per neuron, so feature–neuron correspondence gets cleaner. At 260k width, 93.3% of layer-22 neurons are claimed by at least one feature at 0.7 cosine.

Aggregated across all 13 transcoders, nearly half of late-layer activations come from neuron features. Exceptions exist: Llama 3.2 1B only ~38%; Qwen 14B's middle layers (63%) exceed late layers (53%). The paper doesn't hide these, flagging "mechanism unknown."

There's also a frequency profile: neuron features fire on average 10× more often than other features; at the median, a feature fires at only 0.34× its aligned neuron's rate — features aren't neuron repeaters; they show up in only some of the neuron's contexts. That's exactly what "a feature more specialized than a neuron" looks like.

Three Health Checks: Only Neuron Features Survive

Numbers show "who does the work"; the paper's harsher question is "whose is it — the model's, or training luck?" Three tests, on a self-trained transcoder for layer 22 of Gemma 3 4B:

1. Seed swaps. Retrain the same config with a different random seed and see which features persist. Neuron features mostly survive; non-neuron features largely vanish. 2. Jacobian comparison. Compare the transcoder's input–output derivatives (Jacobian) with the true MLP's, per feature — a hard faithfulness standard. The neuron-feature portion matches the MLP; the total Jacobian similarity over all features is about 0.3, and removing non-neuron features actually raises it. In other words, the dictionary's faithfulness credit is earned mainly by neuron features. 3. Initialization. Every feature starts from a random initialization. Non-neuron features don't move far from their starting points — training barely changed them. They look like initialization noise deposited into the dictionary, not structure the model uses.

Taken together, the paper's phrasing: only neuron features have a stable correspondence to something in the underlying MLP. The remaining features are less faithful, unstable across seeds, and close to random initialization.

A "Red" Interlude

There's a neat case in the appendix. Show the model "красное" (Russian for "red") — presumably a dedicated "red" feature should be present. Inspecting one by one: the transcoder feature at that position is indeed aligned to neurons — but to three polysemantic layer-22 neurons, not some clean "red" unit. The feature dictionary didn't conjure "red" out of thin air; it repackaged the activity of three neurons to make the story easier to tell. Polysemanticity wasn't eliminated, just moved into better-named units.

Is This a Debunking?

No — a calibration. Transcoders remain the strongest MLP stand-ins on reconstruction accuracy (FVU, KL); circuit analysis still works; and dictionaries do deliver things pure neurons can't — the appendix has bigram features, structure absent from neurons. The paper keeps this side in view, not one-sided.

What it moves is the default expectation. The old storytelling order: neurons are too polysemantic, so we learned clean features. This paper reverses it: in learned dictionaries, the stable, reproducible part is mainly the neurons themselves; genuinely "new" non-neuron structure exists too, but it's precisely the part that fails seed swaps and Jacobian checks — to use it well, new attribution methods must be invented first. That's the paper's closing request: non-neuron structure needs proper tools before it can be put to work.

Next time you read a feature-dictionary paper, ask one more question: does this result survive a seed swap? Of the 10,240 neurons in layer 22, 8.9% of the dictionary carries 63.8% of activations — the dictionary wasn't trained in vain, but its most valuable part was copied from the model's own draft.

---

*Source: arXiv 2610.24521v1 (2026-10-17, Shabalin & Piotrowski, Anthropic Fellows × Georgia Tech × Anthropic); the 8.9%/63.8% figures come from Table 4 (Gemma Scope 2 Gemma 3 4B row, late-layer config); the 13-transcoder aggregate, the three seed/Jacobian/initialization checks, 93.3% coverage, the 0.34 frequency ratio, and the "red" case all come from the paper's main text and appendix; the correspondence between the paper's per-layer figure (Figure 19) and Table 4's layer numbers is not stated explicitly in the text, so this post uses "late layers" phrasing.*