The SAE "Dilution" Puzzle: We Thought We Were Looking at Switches, but It's Actually Dials
1. Color Isn't a Switch, It's a Dial
Imagine adjusting the volume on a speaker. Some speakers have dozens of discrete steps: 0, 1, 2, 3... you press once, volume jumps one step. That's discrete.
But some speakers have a smooth dial: you can stop anywhere—2.5, 2.7, 2.71... volume is continuous.
When we talk about an AI model "understanding" a concept—say, "color"—we subconsciously assume the model treats color as a switch: red is one switch, blue is another. Flip the red switch, and the model knows this is red.
But a paper from Harvard, Stanford, Northeastern, and Goodfire tells us: the model actually treats color as a dial.
They ran PCA projections on Llama3.1-8B. When inputs involve continuous concepts—age, temperature, days of the week, colors—the model's activation vectors don't scatter across a few isolated points; instead they lie along smooth curves, surfaces, even rings.
Color is not a discrete label. It's a two-dimensional manifold: one dimension is hue (going around a circle), one is brightness (up and down). What the model "sees" isn't "red = on," but "this point at this position on the color wheel."
That sounds cool. But the problem is: the tools we use to interpret models assume a switch model.
---
2. SAE's "Directional Bias"
SAEs (Sparse Autoencoders) are currently the hottest tool in AI interpretability. Anthropic used them to discover "interpretable features" inside models—like the "Golden Gate" neuron that only fires when the Golden Gate Bridge is mentioned.
The SAE's core assumption comes from the Linear Representation Hypothesis (LRH): every concept corresponds to an independent direction in activation space. All concepts superimpose, like stacking several transparencies, each with a single pattern.
This assumption gave rise to the SAE's mathematical structure:
- Decode with a dictionary D: x ≈ zD
- Sparse code z: only a few nonzero elements
- Each nonzero element corresponds to an "atom" (direction)
- Too many atoms allocated to the same manifold—far more than the manifold's true dimension requires
- Mixed selectivity among these atoms—some are "local detectors" (sensitive only to a small region of the manifold), some are "global blurrers" (weakly responsive to the whole manifold)
- Chaotic coactivation patterns—you can't infer any global structure from a single atom's behavior
- Tiling: atoms are highly selective, each covering a small patch, but together they tile the surface completely
- Capture: a few atoms compactly span the entire subspace; every input activates the same set
- Dilution: many redundant atoms activate, but individual atoms mix selectivity with global fuzziness—the overall structure gets "diluted"
- Stop asking "what concept does this feature represent"
- Start asking "what geometric surface does this group of features jointly cover"
- Stop doing "feature-level interventions"
- Start doing "manifold-level interventions"—moving representations smoothly along the surface
- Title: Do Sparse Autoencoders Capture Concept Manifolds?
- arXiv: 2604.28119
- Authors: Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Owen Lewis, Thomas McGrath, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana
- Institutions: Harvard, Stanford, Northeastern, Goodfire, Technion IIT
- Published: 2026-04-30
If concepts were really switches, this model would be perfect. Each switch (atom) corresponds to a concept; flip it, and the model "activates" the concept.
But what if concepts are dials? What if "red" isn't on/off, but "this position on the color wheel"?
A dial can't be represented by a single switch. You need a group of switches, each covering a small interval of the dial. "Reddish-orange" gets one switch, "crimson" another, "maroon" a third... you combine all these switches' outputs with weights to reconstruct the dial's position.
This is the paper's core finding: SAEs can indeed capture manifolds, but they do so in a fragmented, suboptimal way.
---
3. Two Ways to Capture: Global vs. Local
The paper builds an elegant theoretical framework called "Additive Mixture of Manifolds."
Imagine the model's representation space as a huge room. Each concept occupies a "surface" in the room—maybe a line (1D manifold), a plane (2D manifold), or a ring (like a donut). Different concept surfaces can superimpose, like stacked sheets of curved glass.
The SAE installs a set of "spotlights" (atoms/features) in this room. Each spotlight illuminates a region. The question: how do you illuminate these surfaces?
Approach 1: Global capture (Subspace Capture)
If the concept surface lies in a low-dimensional subspace—say, the color manifold lies in a 2D plane—the SAE can allocate a compact set of atoms whose linear span exactly contains the whole plane.
It's like using three spotlights to illuminate a triangular region. Each light alone covers one corner, but together they light the entire triangle.
Approach 2: Local tiling
What if the concept surface is highly curved and doesn't lie in any low-dimensional subspace? The SAE can adopt a tiling strategy: allocate many atoms, each illuminating only a small local patch of the surface.
It's like lighting a winding mountain road with dozens of small LEDs. Each light covers a meter underfoot. Looking at any one light, you only see "this stretch is gray"—no global meaning. But piece all the lights together and the whole road lights up.
The paper proves an important theorem: under ideal conditions (a sufficiently incoherent dictionary, sparsity matching the manifold's dimension, low enough reconstruction error), an SAE can capture manifolds globally.
But in practice, SAEs rarely meet these ideal conditions.
---
4. "Dilution": The Worst Intermediate State
Experiments reveal a troubling phenomenon the authors call "dilution."
Imagine painting a picture. With a wide brush, one stroke covers a large area—that's the global approach. With a fine-tip marker, each stroke is a thin line, but many lines can compose the picture—that's local tiling.
What is dilution? It's using too many wide brushes AND too many fine markers, allocated with no plan. Some areas get repeatedly smothered by wide brushes (redundancy), some have only a single thin line (gaps), and most areas are a chaotic blend of colors with no discernible pattern.
In SAE terms, dilution means:
Figure 3 in the paper contrasts three regimes:
Experimental data shows that on synthetic benchmarks (with known manifolds embedded in R¹²⁸), SAEs achieve their best global capture around sparsity k=4. But as k increases, the system rapidly slides into the dilution regime—atoms proliferate while each atom's specificity drops.
---
5. Why Individual Features Are Unreliable
This explains many frustrating "negative results" in the SAE field:
1. Dictionary instability
Train an SAE on the same model twice, and you get completely different features. Under the switch model, this is puzzling: the red switch should always be the red switch. Under the manifold model, it's natural: the same surface can be tiled in different ways. Two training runs are like two workers tiling the same floor with different tiles—the result works either way, but the layouts differ.
2. Individual feature steering fails
If you find a "Wednesday" feature and artificially increase its value, expecting more Wednesday-related output, the model's behavior often becomes chaotic—not more Wednesday, but degenerate, meaningless text.
Why? If the feature is just a local detector on the Wednesdays manifold, increasing its value pushes the model's representation off the manifold—like yanking the steering wheel in a moving car: the car doesn't go where you want; it veers off the road.
3. Difficulty of automated interpretability
The accuracy of automatically labeling what each SAE feature "represents" has consistently underwhelmed. Why? A single feature doesn't "represent" a full concept in the first place. A concept is the whole manifold, the collective behavior of a group of features. It's like judging an entire painting from a single pixel—impossible.
---
6. Looking for "Feature Groups," Not "Features"
The paper's solution is elegant: don't look for single features—look for feature groups.
They propose an unsupervised method based on the Ising model. The idea: if a group of features always activates together (or stays silent together), they may be cooperating to cover the same manifold.
The method: 1. Binarize SAE encodings (active/inactive) 2. Fit a pairwise Ising model to learn a coupling matrix J between features 3. Find block-diagonal structure in J—clusters of strongly positively or negatively correlated features 4. These clusters are "candidate manifold detectors"
On synthetic data, this method perfectly recovers ground-truth manifold partitions. On real LLMs, it also finds manifold structures confirmable by human annotation (like the circular manifold of days of the week).
But the authors admit limitations: if dilution is too severe, the coupling signal between features gets drowned in noise. And Ising models only capture pairwise relationships—more complex higher-order interactions need stronger tools.
---
7. The Bigger Picture
This paper raises a more fundamental question:
Have we been looking at models with the wrong microscope all along?
SAEs are popular because they decompose complex neural representations into "human-readable" switches. Each switch has a label: "Golden Gate," "Wednesday," "anger." It's intuitive and fits our cognitive habits.
But if the model's internal representations are really geometric objects—manifolds, surfaces, rings—then the "switch" description is fundamentally wrong. It's like a doctor using a stethoscope to examine an X-ray—the tool doesn't match the target.
The paper's conclusion: future representation-learning methods should treat geometric objects (manifolds), not just independent directions, as the basic unit of interpretability.
What does that mean? It means we need entirely new tools:
---
Paper info