The "Dilution" Mystery in SAEs: We Thought We Were Looking at Switches, but It's Actually Knobs
A walkthrough of "Do Sparse Autoencoders Capture Concept Manifolds?" (arXiv: 2604.28119), by researchers from Harvard, Stanford, Northeastern, Goodfire, and Technion IIT.
1. Concepts Are Knobs, Not Switches
Imagine adjusting volume on a speaker. Some speakers have discrete steps: 0, 1, 2, 3... One press, one step. Others have a smooth knob: you can stop anywhere—2.5, 2.7, 2.71.
When we talk about an AI model "understanding" a concept like "color," we subconsciously assume the model treats color as a switch—red is one switch, blue is another. Flip the red switch, and the model knows it's red.
But this paper shows that models actually treat color as a knob. Applying PCA projections to Llama3.1-8B activations, the authors found that for continuous concepts—age, temperature, day of the week, color—the activation vectors don't sit at isolated points but lie along smooth curves, surfaces, and even rings.
Color is not a discrete label. It is a two-dimensional manifold: one dimension for hue (around a circle), one for brightness (up and down). The model doesn't see "red = on"; it sees "this point at this position on the color wheel."
The problem: our interpretability tools assume the switch model.
2. SAE's "Direction Bias"
Sparse Autoencoders (SAEs) are the hottest tool in AI interpretability. Anthropic used them to discover "interpretable features"—like the famous Golden Gate feature that only activates when the Golden Gate Bridge is mentioned.
SAEs rest on the Linear Representation Hypothesis (LRH): each concept corresponds to an independent direction in activation space. Concepts superimpose like stacked transparencies, each holding one pattern.
This yields the SAE's mathematical structure:
- Decode with a dictionary D: x ≈ zD
- Sparse code z: only a few nonzero entries
- Each nonzero entry corresponds to an "atom" (direction)
- Too many atoms assigned to the same manifold—far more than its true dimension
- Mixed selectivity—some atoms are "local detectors," others are "global blurrers" with weak response to the entire manifold
- Chaotic co-activation patterns—no global structure inferable from a single atom
- Tiling: highly selective atoms, each covering a small patch that assembles into complete coverage
- Capture: a few compact atoms spanning the whole subspace, activated together
- Dilution: many redundant atoms with mixed selectivity—the overall structure is "diluted"
- Stop asking "what concept does this feature represent?"
- Start asking "what geometric surface does this feature group jointly cover?"
- Stop doing feature-level interventions
- Start doing manifold-level interventions—moving representations smoothly along the surface
- Title: Do Sparse Autoencoders Capture Concept Manifolds?
- arXiv: 2604.28119
- Authors: Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Owen Lewis, Thomas McGrath, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana
- Institutions: Harvard, Stanford, Northeastern, Goodfire, Technion IIT
- Released: 2026-04-30
If concepts were switches, this would be perfect. But what if "red" isn't on/off—just a position on the color wheel?
A knob can't be represented by a single switch. You need a group of switches, each covering a small interval of the knob. You combine their outputs to reconstruct the knob's position.
The paper's core finding: SAEs can capture manifolds, but they do so in a fragmented, suboptimal way.
3. Two Capture Modes: Global vs. Local
The paper builds a framework called "Additive Mixture of Manifolds." Picture the model's representation space as a large room where each concept occupies a "surface"—a line (1D manifold), a sheet (2D), or a ring. SAE installs a set of "searchlights" (atoms). How should they illuminate these surfaces?
Mode 1: Subspace Capture. If a concept's manifold lies in a low-dimensional subspace, SAE can assign a compact set of atoms whose linear span contains the whole plane—like three searchlights jointly covering a triangular region.
Mode 2: Tiling. If the manifold is too curved to lie in a low-dimensional subspace, SAE spreads many atoms, each covering a small local patch—like dozens of small LEDs lighting a winding mountain road. A single LED tells you nothing about the whole path.
The paper proves an important theorem: under ideal conditions (sufficiently incoherent dictionary, sparsity matched to manifold dimension, low reconstruction error), SAEs can capture manifolds globally.
But real SAEs rarely meet these conditions.
4. "Dilution": The Worst Middle Ground
Experiments reveal a troubling phenomenon the authors call "dilution."
If global capture is painting with a broad brush, and tiling is tracing with a fine marker, dilution is using too many of both, randomly. Some areas are redundantly over-painted; some are barely covered; most are a muddy mess.
In SAE terms, dilution means:
Figure 3 in the paper contrasts three regimes:
On synthetic benchmarks (known manifolds embedded in R¹²⁸), SAEs achieve their best global capture around sparsity k=4. As k grows, the system quickly slides into the dilution regime—atom count balloons while per-atom specificity drops.
5. Why Single Features Are Unreliable
This explains the field's frustrating "negative findings":
1. Dictionary instability. Train the same SAE twice and you get completely different features. Under the switch model this is confusing—but under the manifold model it's natural: the same surface can be tiled many ways, like two workers laying different tiles over the same floor.
2. Feature steering fails. Amplify a "Wednesday" feature and expect more Wednesday content—instead, model behavior often degenerates into nonsense. Why? If the feature is just a local detector on the Wednesdays manifold, amplifying it pushes the representation off the manifold—like yanking the steering wheel so hard the car leaves the road entirely.
3. Automated interpretability struggles. Auto-labeling individual SAE features has low accuracy because a single feature doesn't "represent" a concept. The concept is the whole manifold—the collective behavior of a group of features. Judging a painting from one pixel is impossible.
6. Finding "Feature Groups," Not "Features"
The paper's solution: find groups of features.
They propose an unsupervised method based on the Ising model: 1. Binarize SAE encodings (active/inactive) 2. Fit a pairwise Ising model to learn a coupling matrix J between features 3. Find block-diagonal structure in J—clusters of strongly (anti-)correlated features 4. These clusters are "candidate manifold detectors"
On synthetic data, the method perfectly recovers ground-truth manifold partitions. On real LLMs, it finds manifold structures confirmed by human annotation (e.g., the cyclical manifold of weekdays).
Limitations: severe dilution drowns coupling signals in noise, and Ising models only capture pairwise relations—higher-order interactions need stronger tools.
7. The Bigger Picture
Have we been looking at models through the wrong microscope all along?
SAEs are popular because they decompose neural representations into human-readable "switches," each with a label. Intuitive—but if the model's internal representations are truly geometric objects (manifolds, surfaces, rings), switch-based description is fundamentally wrong, like examining an X-ray with a stethoscope.
The paper concludes that future representation-learning methods should treat geometric objects (manifolds), not just independent directions, as the basic unit of interpretability:
---
Paper info