Does AI Really Understand "Dog"? When Concepts Are Manifolds, Not Directions
> *Anthropic's scientists recently did something remarkable: they opened up the brain of a production-grade AI and found a dedicated "sycophantic praise" feature. When Claude reads "he is a generous and kind person," this feature lights up intensely. But this raises a deeper question: does this feature truly "understand" praise, or has it merely learned to light up under specific word-vector combinations?*
---
1. A Philosophical Crisis Sparked by a Sycophancy Feature
In May 2024, Anthropic published a paper that shook the AI world. Using a technique called Sparse Autoencoders (SAEs), they mined millions of interpretable features from the "brain" of their production model Claude 3 Sonnet.
Some of these features are spine-chilling:
- A "human flaws" feature that activates on "My dad wasn't perfect (but who is?) but he loved us deeply"
- A "deception and power-seeking" feature
- A "sycophantic praise" feature
- Some features attempt global representation but cover it incompletely
- Others attempt local tiling, but the patches connect incoherently
- The result: the manifold's structure is "diluted" across many features, with no clear manifold pattern visible in any single feature
It sounds like AI really has ideas. But in April 2026, a team of 12 researchers including Usha Bhalla, Thomas Fel, and Can Rager published a paper so calm it was nearly brutal, throwing cold water on this celebration.
They found: we may have fundamentally misunderstood how AI represents concepts. If concepts are not discrete directions but continuous manifolds—then SAEs, the much-hyped "AI microscope," may be showing us the wrong picture.
---
2. Opening the Black Box: A Decade-Long Journey in Mechanistic Interpretability
To understand the weight of this problem, we need to return to the origins of deep learning.
In 2012, AlexNet won the ImageNet competition by an overwhelming margin, officially opening the deep learning era. But almost from day one, researchers faced an awkward predicament: these models worked extremely well, but nobody knew why.
A neural network may have billions of parameters, each a tiny number optimized by gradient descent. Together, after training on trillions of tokens, they can write poetry, solve math problems, and even pass the bar exam. But ask it "how did you do it?" and you get only silence.
Early interpretability efforts focused on feature visualization. A team led by Chris Olah at Google Brain developed a technique: using gradient ascent to generate images that maximally activate a given neuron. Applying this to Inception network neurons revealed surprising patterns—some neurons were sensitive to "dog ears," others to "spiral textures."
But this method has a fatal flaw: individual neurons are often "polysemantic". The same neuron may respond to academic citations, English conversation, HTTP requests, and Korean text. You can't point at it and say "this is the dog detector."
In 2017, a famous word2vec example gave new hope: if you take the vector for "king," subtract "man," and add "woman," the result is nearly exactly the vector for "queen." This suggested a seductive hypothesis—concepts are represented as directions in a neural network's activation space.
Thus was born the Linear Representation Hypothesis. Its core claim is extremely concise: there exists a direction vector d such that the projection of activation vector a onto d measures the strength of that concept.
If this hypothesis holds, understanding neural networks becomes a geometry problem: find all the directions corresponding to concepts, and you have AI's "vocabulary."
---
3. Superposition: Cheating Magic in High-Dimensional Space
But the linear representation hypothesis faces a fundamental mathematical obstacle.
Suppose a network layer has n = 100,000 neurons. Under the linear representation hypothesis, it can represent at most 100,000 independent linear directions. But models like GPT-4 clearly "know" far more than 100,000 concepts—"quantum entanglement," "French onion soup," "17th-century Dutch still-life painting"—millions or even billions of concepts.
How does a neural network manage this?
In 2022, Anthropic's Nelson Elhage, Chris Olah, and colleagues published "Toy Models of Superposition," revealing the answer: neural networks cheat in high-dimensional space.
In n-dimensional space, there can be at most n orthogonal vectors. But math has a stunning fact: you can have exponentially many "nearly orthogonal" vectors. The Johnson-Lindenstrauss lemma tells us that in high-dimensional space, you can pack roughly exp(n) vectors that are all nearly 90 degrees apart.
Specifically, if two vectors are close enough to perpendicular (cosine similarity below some ε), then when one vector activates, its projection onto the other is negligible—like a nearly vertical pole casting almost no shadow on the floor.
Neural networks exploit this, encoding millions of features as nearly orthogonal directions. Each feature activates with only negligible "interference" on others. And since real-world concepts are sparse (a text won't simultaneously contain "quantum entanglement" and "French onion soup"), this interference rarely causes actual problems.
This is the Superposition Hypothesis: a neural network uses n-dimensional space to "simulate" a far larger sparse network where each idealized neuron represents an independent feature. From the perspective of individual neurons, this manifests as polysemanticity—each real neuron is a linear combination of multiple features.
This finding explains why single neurons are hard to interpret, and hints at a possible solution: if we could recover those hidden "ideal neurons" from superposition, couldn't we reconstruct AI's true "vocabulary"?
---
4. Sparse Autoencoders: AI's Rosetta Stone?
Sparse autoencoders (SAEs) were born to solve this problem.
An autoencoder is a neural network that compresses input into a "hidden layer," then reconstructs the original input from it. Traditional autoencoders are used for dimensionality reduction (like PCA), with hidden layers usually smaller than the input.
SAEs make a seemingly crazy but actually ingenious design choice: the hidden layer is much larger than the input—typically 4 to 32 times larger. But they also impose a strict sparsity constraint: at any moment, only a tiny fraction of hidden neurons may activate.
It's like equipping AI with a super-dictionary. An ordinary network has only 100,000 "entries" (neurons), while the SAE provides millions of entries—with the rule that each "utterance" may use only a few of them.
By training the SAE to reconstruct the activations of a target network layer, researchers hope it learns to "un-superpose"—decomposing entangled features into independent, interpretable units.
In 2023, Anthropic first demonstrated SAE effectiveness on small transformers. In 2024, they scaled SAEs to the production model Claude 3 Sonnet, extracting millions of features. OpenAI and Google DeepMind followed with similar work on their own models.
These findings are exciting, but they all rest on an implicit assumption: concepts correspond to independent linear directions. Each SAE feature is a direction vector; when input contains a concept, activation along that direction increases.
But is this assumption actually true?
---
5. Concepts Are Manifolds, Not Directions
Bhalla et al.'s paper borrows a deep insight from cognitive science and neuroscience: many concepts may not be discrete directions but continuous manifolds.
What is a manifold? Imagine a globe. Mathematically, Earth's surface is a 2D manifold—locally, at every point's neighborhood, it looks like a flat plane (you can unfold a map), but globally it has curved topology.
Color is a classic manifold concept. Color is not discrete points ("red," "orange," "yellow") but a continuous spectrum. In human perceptual space, colors form a ring-like manifold: red to orange to yellow to green to blue to purple, with purple looping back to red through magenta.
Emotions are similar. Psychologists have found that human emotional space can be described by a 2D manifold: one axis is "valence" (pleasant vs. unpleasant), the other "arousal" (excited vs. calm). Anger, sadness, joy, fear—these are not isolated islands but different positions on a continuous landscape.
In 2022, a Stanford team led by Surya Ganguli, in a study published in *Science Advances*, directly measured the geometric structure of concepts in macaque inferior temporal (IT) cortex. They showed monkeys images of 64 different concepts (bears, cows, dogs, cars, etc.) while recording activity from 168 IT neurons.
The result: each concept formed a low-dimensional manifold in neural activity space—specifically, an ellipsoid of roughly 35 dimensions embedded in the 2048-dimensional global space. More importantly, semantic relationships between concepts mapped directly onto geometric relationships between manifolds: semantically close concepts (like bear and wolf) had manifolds closer together in neural space, while semantically distant concepts (like bear and car) had farther-apart manifolds.
If concepts in the brain are manifolds, then concepts in AI are likely manifolds too—after all, the transformer architecture was partly inspired by imitation of the brain's information processing.
---
6. Experimental Revelation: SAEs' "Dilution" Dilemma When Facing Manifolds
Bhalla et al. posed three core questions: 1. What does it mean for an SAE to capture a manifold? 2. When can existing SAE architectures do so? 3. How can they do so?
They developed a rigorous theoretical framework and found SAEs can capture manifolds in two fundamentally different ways:
Way 1: Global capture. The SAE allocates a compact set of atoms (features) whose linear span contains the entire manifold. Imagine approximating a circle with a set of lines—if there are enough of them, well distributed, their envelope approximates the circle well.
Way 2: Local capture. The SAE distributes the manifold across many features, each covering only a local region. Imagine tiling a sphere with small tiles—each tile covers only a small patch of curved surface.
Theoretically these two ways are equivalent; both can exactly represent the manifold. But Bhalla et al.'s experiments revealed a disturbing fact: real-world SAEs do neither global nor local capture well, but fall into an intermediate state they call "dilution".
In the dilution state:
Bhalla et al. also found a deeper implication: if concepts really are manifolds, then existing SAE architectures—and the entire "direction"-based interpretability paradigm—may need fundamental reconstruction. Future representation learning approaches should treat geometric objects (manifolds, subspaces, topological structures), not just individual directions, as the basic units of interpretability.
---
7. From Directions to Manifolds: A Paradigm Shift in Interpretability
What do Bhalla et al.'s findings mean?
First, they remind us: SAEs are not an all-purpose AI microscope. They are a very powerful tool, but they rest on specific mathematical assumptions. If those assumptions don't match AI's true representational structure, SAEs give a misleading picture—like a microscope designed to observe particles being used to observe waves.
Second, they suggest the nature of concepts may be more complex than the linear representation hypothesis. Concepts in human cognition clearly have continuity and geometric structure. "Red" is not a binary red/non-red switch but a continuous spectrum. "Anger" is not an on/off boolean but an emotional dimension varying continuously from "mild irritation" to "fury."
If AI also uses manifolds to represent concepts, this is actually good news: it means AI's representational structure may be closer to human cognition than we thought.
But it also poses a challenge: we need new tools to study manifolds. Bhalla et al. suggest that rather than searching for isolated feature directions, we should search for coherent groups of atoms—feature sets that jointly cover some region of a manifold. This resembles the concept of population coding in neuroscience: what matters is not the activity of single neurons but the activity patterns of neuronal populations.
Finally, this touches a deeper philosophical question: does AI really "understand" concepts?
If concepts are just directions in high-dimensional space, "understanding" reduces to a geometric relation—concept A near concept B, concept C in the opposite direction. But if concepts are manifolds, "understanding" involves grasping the manifold's geometry: knowing how to navigate on it, knowing what continuous variations in local neighborhoods mean.
This recalls mathematician Henri Poincaré's remark: "Geometry is the art of reasoning about a group of operations." Perhaps understanding a concept also means mastering a group of operations—knowing how to move smoothly from one concept to another, knowing which changes are "small" and which are "fundamental."
---
8. Conclusion
Bhalla et al.'s paper does not negate the value of SAEs. SAEs remain one of the most powerful interpretability tools today. But it reminds us: tools are not truth.
When we use SAEs to observe AI's brain, what we see is an image processed through the SAE "filter." If AI's true conceptual structure is a manifold, and the SAE filter was designed for directions, then what we see will always be a distorted image.
But this is precisely the most beautiful part of science: every study that answers a new question also reveals new ones. SAEs gave us our first glimpse of AI's internal structure, and Bhalla et al.'s work tells us: the structure we see may be richer, more continuous, and more geometric than we imagined.
Does AI really understand "dog"? Perhaps the answer depends on which level you ask. At the level of directions, there may be a "dog direction"; at the level of manifolds, there may be a "dog manifold," covered with continuous variations from Chihuahua to Tibetan Mastiff.
True understanding, perhaps, is not standing at a point, but being able to wander freely across the manifold.
---
References
1. Bhalla, U., Fel, T., Rager, C. et al. *Do Sparse Autoencoders Capture Concept Manifolds?* arXiv:2604.28119 [cs.LG] (2026). 2. Elhage, N. et al. *Toy Models of Superposition.* Transformer Circuits Thread (2022). 3. Bricken, T. et al. *Towards Monosemanticity: Decomposing Language Models With Dictionary Learning.* Anthropic (2023). 4. Templeton, A. et al. *Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet.* Anthropic (2024). 5. Gao, L. et al. *Scaling and Evaluating Sparse Autoencoders.* OpenAI (2024). 6. Chung, Y. et al. *Scaling in Representation Learning.* *Nature Communications* 14, 5108 (2023). 7. Abdelnour, F., Voss, H.U. & Raj, A. *From sensory to perceptual manifolds: The twist of neural geometry.* *Science Advances* 11, eadv0431 (2025). 8. Chung, S. et al. *Neural representational geometry underlies few-shot concept learning.* *PNAS* 119, e2205650119 (2022). 9. Mikolov, T., Yih, W. & Zweig, G. *Linguistic Regularities in Continuous Space Word Representations.* *NAACL-HLT* (2013). 10. Olah, C. et al. *Feature Visualization.* *Distill* (2017).