The SAE "Dilution" Puzzle: We Thought We Were Looking at Switches, but It's Actually Knobs
1. Color Is Not a Switch, It's a Knob
Imagine adjusting the volume on a speaker. Some speakers have dozens of discrete steps: 0, 1, 2, 3... press once, the volume jumps one step. That's discrete.
But some speakers have a smooth knob: you can stop anywhere—2.5, 2.7, 2.71... The volume is continuous.
When we talk about an AI model "understanding" a concept—say, "color"—we subconsciously assume the model treats color as a switch: red is one switch, blue is another. Flip the red switch on, and the model knows it's red.
But this paper from Harvard, Stanford, Northeastern, and Goodfire tells us: models actually treat color as a knob.
They performed PCA projections on Llama3.1-8B. When inputs involve continuous concepts—age, temperature, days of the week, colors—the model's activation vectors don't scatter across a few isolated points. Instead, they lie along smooth curves, surfaces, even rings.
Color is not a discrete label. It is a two-dimensional manifold: one dimension is hue (wrapping around a circle), the other is brightness (up and down). What the model "sees" is not "red = on," but "this point sits at this position on the color wheel."
That sounds cool. But here's the problem: the tools we use to interpret models assume the switch model.
---
2. SAE's "Directional Bias"
SAEs (Sparse Autoencoders) are currently the hottest tool in AI interpretability. Anthropic used them to discover "interpretable features" in models—like the famous "Golden Gate" feature that only activates when the Golden Gate Bridge is mentioned.
The core assumption behind SAEs comes from the Linear Representation Hypothesis (LRH): each concept corresponds to an independent direction in activation space. Concepts stack together like several transparencies overlaid, each carrying a single pattern.
This assumption gives SAEs their mathematical structure:
- Decode with a dictionary D: x ≈ zD
- Sparse code z: only a few nonzero elements
- Each nonzero element corresponds to an "atom" (a direction)
- Too many atoms are allocated to the same manifold—far more than the manifold's true dimensionality
- These atoms have mixed selectivity—some are "local detectors" (sensitive only to a small region of the manifold), others are "global blur-ers" (weakly responsive to the entire manifold)
- Coactivation patterns are chaotic—you can't infer any global structure from a single atom's behavior
- Tiling: atoms are highly selective, each covering a small patch, but together the coverage is complete
- Capture: a small set of atoms compactly spans the entire subspace; every input activates the same group
- Dilution: many redundant atoms activate, but each atom mixes selectivity with global vagueness—the overall structure gets "diluted"
- Stop asking "what concept does this feature represent"
- Instead ask "what geometric surface does this group of features jointly cover"
- Stop doing "feature-level interventions"
- Instead do "manifold-level interventions"—moving representations smoothly along the surface
- Title: Do Sparse Autoencoders Capture Concept Manifolds?
- arXiv: 2604.28119
- Authors: Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Owen Lewis, Thomas McGrath, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana
- Institutions: Harvard, Stanford, Northeastern, Goodfire, Technion IIT
- Released: 2026-04-30
If concepts really were switches, this model would be perfect. Each switch (atom) corresponds to a concept; flip it, and the model "activates" the concept.
But what if concepts are knobs? What if "red" isn't on/off, but "this position on the color wheel"?
A knob can't be represented by a single switch. You need a set of switches, each covering a small interval of the knob. "Reddish-orange" maps to one switch, "crimson" to another, "maroon" to a third... Only a weighted combination of all these switches can reconstruct the knob's position.
This is the paper's core finding: SAEs can indeed capture manifolds, but they do so in a fragmented, suboptimal way.
---
3. Two Capture Modes: Global vs. Local
The paper builds an elegant theoretical framework called "Additive Mixture of Manifolds."
Imagine the model's representation space as a huge room. Each concept occupies a "surface" in the room—a line (1D manifold), a plane (2D manifold), or a ring (like a donut). Different concept surfaces can stack on top of each other, like several curved sheets of glass.
SAEs install a set of "spotlights" (atoms/features) in this room. Each spotlight illuminates one region. The question is: how should these surfaces be illuminated?
Mode 1: Global capture (Subspace Capture)
If the concept surface lies in a low-dimensional subspace—for example, the color manifold lies in a 2D plane—then an SAE can allocate a compact set of atoms whose linear span exactly contains the entire plane.
It's like using three spotlights to illuminate a triangular region. Each light alone covers one corner, but together they light up the whole triangle.
Mode 2: Local tiling (Tiling)
What if the concept surface is highly curved and doesn't lie in any low-dimensional subspace? Then the SAE can use a tiling strategy: allocate many atoms, each illuminating only a small local patch of the surface.
It's like using dozens of small LEDs to light a winding mountain road. Each light covers one meter of ground. Looking at a single light, you only see "this stretch of road is gray"—no global meaning. But piecing all the lights together, the whole road lights up.
The paper proves an important theorem: under ideal conditions (a sufficiently incoherent dictionary, sparsity matching the manifold's dimension, low enough reconstruction error), an SAE can capture manifolds globally.
But in reality, SAEs rarely meet these ideal conditions.
---
4. "Dilution": The Worst Intermediate State
Experiments reveal a troubling phenomenon the authors call "dilution."
Imagine painting a picture. With a broad brush, one stroke covers a large area—global capture. With a fine-tipped marker, one stroke draws a thin line, but many lines together form the whole picture—local tiling.
What is dilution? It's using too many broad brushes and too many fine markers at once, with no coherent allocation. Some areas get repeatedly painted by piles of broad brushes (redundancy); some have only a single thin line (omission); and most areas are a chaotic mess of overlapping colors—you can't see any clear pattern.
In the SAE context, dilution means:
Figure 3 in the paper compares three regimes:
Experimental data on synthetic benchmarks (with known manifolds embedded in R¹²⁸) shows SAEs achieve the best global capture around sparsity k=4. But as k increases, the system rapidly slides into the dilution regime—atom counts balloon while each atom's specificity drops.
---
5. Why Are Individual Features Unreliable?
This explains why the SAE field has seen so many frustrating "negative results":
1. Dictionary instability
Train an SAE twice on the same model and the learned features are completely different. Under the switch model, this is puzzling: the red switch should always be the red switch. Under the manifold model, it's natural: the same surface can be tiled in many different ways. Two training runs are like two workers tiling the same floor with different tiles—the result is equivalent, but the tile layouts differ.
2. Single-feature steering doesn't work
If you find a "Wednesday" feature and artificially increase its value, hoping the model outputs more Wednesday-related content, the result is often that the model's behavior becomes chaotic—not more Wednesday-like, but degraded into meaningless text.
Why? If the feature is merely a local detector on the Wednesdays manifold, increasing its value pushes the model's representation off the manifold—like yanking the steering wheel hard in a moving car: instead of going where you want, the car flies off the road.
3. Automated interpretability struggles
Accuracy for automatically labeling what each SAE feature "represents" has stayed low. Why? A single feature simply doesn't "represent" a complete concept. The concept is the entire manifold—the collective behavior of a group of features. It's like judging a whole painting from a single pixel—impossible.
---
6. Looking for "Feature Groups," Not "Features"
The paper's solution is elegant: don't look for individual features; look for feature groups.
They propose an unsupervised method based on the Ising model. The core idea: if a set of features always activates together (or stays silent together), they may be "cooperating" to cover the same manifold.
The procedure: 1. Binarize the SAE encodings (active/inactive) 2. Fit a pairwise Ising model to learn the coupling matrix J between features 3. Find "block-diagonal structure" in J—clusters of strongly positively or negatively correlated features 4. These clusters are "candidate manifold detectors"
On synthetic data, this method perfectly recovers the ground-truth manifold partitioning. On real LLMs, it also discovers manifold structures confirmable by human annotation (such as the circular manifold of days of the week).
But the authors admit limitations: if an SAE's dilution is too severe, the coupling signal between features gets drowned in noise. And the Ising model only captures pairwise relationships—more complex higher-order interactions will need stronger tools.
---
7. The Bigger Picture
This paper raises a more fundamental question:
Have we been looking at models with the wrong "microscope" all along?
SAEs are popular because they decompose complex neural representations into "human-readable" switches. Each switch has a label: "Golden Gate," "Wednesday," "anger." This is intuitive and matches our cognitive habits.
But if the model's internal representations are really geometric objects—manifolds, surfaces, rings—then the "switch" framing is fundamentally wrong. It's like a doctor using a stethoscope to examine an X-ray—a mismatch between tool and target.
The paper's conclusion: future representation-learning methods should treat geometric objects (manifolds), not just independent directions, as the basic unit of interpretability.
What does that mean? It means we need new tools:
---
Paper information