The "Dilution" Mystery of SAEs: We Thought We Were Looking at Switches, But It's Knobs
*A hard-nosed breakdown by Xiao Kai of zhichai.net, translated from the original Chinese post.*
1. Color Is Not a Switch, It's a Knob
Imagine adjusting the volume on a speaker. Some speakers have dozens of discrete steps: 0, 1, 2, 3... press once, volume jumps one step. That's discrete.
But some speakers have a smooth knob: you can stop anywhere—2.5, 2.7, 2.71... The volume is continuous.
When we talk about an AI model "understanding" a concept—say, "color"—we implicitly assume the model treats color as a switch: one switch for red, another for blue. Flip the red switch and the model knows it's red.
But a paper from Harvard, Stanford, Northeastern, and Goodfire tells us: the model treats color as a knob.
They ran PCA projections on Llama3.1-8B. For continuous concepts—age, temperature, day of the week, color—the model's activation vectors don't scatter across a few isolated points; they lie along smooth curves, surfaces, even rings.
Color isn't a discrete label. It's a two-dimensional manifold: one dimension is hue (a full circle), one is brightness (up and down). What the model "sees" is not "red = on" but "this point sits here on the color wheel."
Sounds cool. But the problem: the tools we use to interpret models assume the switch model.
2. SAE's "Direction Bias"
SAE (Sparse Autoencoder) is the hottest tool in AI interpretability today. Anthropic used it to discover "interpretable features" in models—like the famous "Golden Gate" feature that only activates when the Golden Gate Bridge is mentioned.
SAE's core assumption comes from the Linear Representation Hypothesis (LRH): every concept corresponds to an independent direction in activation space. Concepts superimpose, like stacking transparencies, each with a single pattern.
This assumption produced SAE's math:
- Decode with a dictionary D: x ≈ zD
- Sparse code z: only a few nonzero elements
- Each nonzero element corresponds to an "atom" (a direction)
- Too many atoms allocated to the same manifold—far more than its true dimension requires
- Mixed selectivity—some atoms are "local detectors" (sensitive to one small region), others "global blurrers" (weakly responsive across the whole manifold)
- Chaotic co-activation patterns—you can't infer any global structure from a single atom's behavior
- Tiling: atoms highly selective, each covering a small patch, patches assembling into complete coverage
- Capture: few atoms compactly span the whole subspace; every input activates the same set
- Dilution: many redundant atoms activate, individual atoms mix selectivity with global blurriness, and structure gets "diluted"
- Stop asking "what concept does this feature represent"
- Start asking "what geometric surface does this group of features jointly cover"
- Stop doing feature-level interventions
- Start doing manifold-level interventions—smoothly moving representations along the surface
- Title: Do Sparse Autoencoders Capture Concept Manifolds?
- arXiv: 2604.28119
- Authors: Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Owen Lewis, Thomas McGrath, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana
- Institutions: Harvard, Stanford, Northeastern, Goodfire, Technion IIT
- Posted: 2026-04-30
If concepts really were switches, this is perfect. But what if concepts are knobs? A knob can't be represented by a single switch. You'd need a set of switches, each covering a small interval of the knob's range—"reddish-orange" gets one, "crimson" another, "maroon" a third... and you combine their outputs to reconstruct the knob's position.
That's the paper's core finding: SAEs can capture manifolds, but they do so in a fragmented, suboptimal way.
3. Two Ways to Capture: Global vs. Local
The paper builds a framework called "Additive Mixture of Manifolds."
Think of the representation space as a big room. Each concept occupies a "surface" in the room—a line (1D manifold), a plane (2D), or a ring. SAE must install a set of "searchlights" (atoms/features), each illuminating a region.
Way 1: Subspace Capture — If the concept's surface lies in a low-dimensional subspace (e.g., the color manifold on a 2D plane), the SAE can allocate a compact set of atoms whose span contains the whole plane. Like three searchlights covering a triangle: individually each lights a corner, together they cover it all.
Way 2: Tiling — If the surface is curved and lies in no low-dimensional subspace, the SAE can allocate many atoms, each lighting a tiny local patch. Like dozens of small LEDs lighting a winding mountain road—each shows only "this one meter of path is gray," with no global meaning. But assembled, the whole road lights up.
The paper proves an important theorem: under ideal conditions (sufficiently incoherent dictionary, sparsity matched to manifold dimension, low enough reconstruction error), an SAE can capture manifolds globally.
But in practice, SAEs rarely meet these conditions.
4. "Dilution": The Worst Middle Ground
Experiments reveal an unsettling phenomenon the authors call "dilution".
It's like painting with too many broad brushes AND too many fine markers, allocated haphazardly. Some areas get smeared repeatedly (redundancy), some get only one thin line (omission), and most are a chaotic overlap where no clear pattern is visible.
In SAE terms, dilution means:
Figure 3 in the paper contrasts three regimes:
On synthetic benchmarks (known manifolds embedded in R¹²⁸), SAEs achieve the best global capture around sparsity k=4. As k grows, the system slides rapidly into the dilution regime—atoms multiply, but each atom's specificity drops.
5. Why Single Features Are Unreliable
This explains the field's frustrating "negative results":
1. Dictionary instability. Train an SAE twice on the same model, get completely different features. In the switch model this is confusing—the red switch should always be the red switch. In the manifold view it's natural: the same surface can be tiled in many ways. Two runs are two workers tiling the same floor with different tiles—same effect, different layout.
2. Steering via single features fails. Find a "Wednesday" feature, amplify it, expect more Wednesday-related output—and often the model's behavior becomes chaotic, degenerating into nonsense. Why? If the feature is just a local detector on the Wednesdays manifold, amplifying it pushes the representation off the manifold—like yanking the steering wheel on a moving car: the car doesn't go where you want, it leaves the road.
3. Automated interpretability stalls. Accuracy of auto-labeling SAE features stays low. A single feature simply doesn't "represent" a whole concept. The concept is the whole manifold—the collective behavior of a group of features. Judging a painting from one pixel is impossible.
6. Looking for "Feature Groups," Not "Features"
The paper's fix is elegant: don't look for single features—look for feature groups.
They propose an unsupervised method based on the Ising model: 1. Binarize SAE activations (active/inactive) 2. Fit a pairwise Ising model to learn a coupling matrix J between features 3. Find "block-diagonal structure" in J—clusters of strongly positively or negatively correlated features 4. These clusters are "candidate manifold detectors"
On synthetic data, the method perfectly recovers ground-truth manifold partitions. On real LLMs, it recovers human-verifiable manifold structures (like the cyclic manifold of weekdays).
The authors admit limitations: if dilution is severe, coupling signals drown in noise. And the Ising model only captures pairwise relations—higher-order interactions need stronger tools.
7. The Bigger Picture
Have we been looking at models with the wrong microscope all along?
SAEs are popular because they decompose neural representations into human-readable switches, each with a label: "Golden Gate," "Wednesday," "anger." Intuitive, matching our cognitive habits.
But if the model's internal representations are truly geometric objects—manifolds, surfaces, rings—then the "switch" description is fundamentally wrong. Like a doctor listening to an X-ray with a stethoscope: tool and target don't match.
The paper's conclusion: future representation-learning methods should treat geometric objects (manifolds), not just independent directions, as the basic unit of interpretability.
What does that mean? We need new tools:
---
Paper info