Does AI Really Understand "Dog"? When Concepts Are Manifolds, Not Directions

> *Anthropic scientists recently did something remarkable: they opened the brain of a production-grade AI and found a dedicated "sycophantic praise" neuron. When Claude reads "he is a generous and kind person," this feature fires intensely. But this raises a deeper question: does this feature really "understand" praise, or has it merely learned to light up under specific word-vector combinations?*

---

A philosophical crisis sparked by a "sycophancy feature"

In May 2024, Anthropic published a landmark paper. Using sparse autoencoders (SAEs), they mined millions of interpretable features from the "brain" of their production model Claude 3 Sonnet.

Some findings were chilling:

• A "humans have flaws" feature, activating on "My dad wasn't perfect (who is?) but he loved us deeply"
• A "deception and power-seeking" feature
• A "sycophantic praise" feature

It sounded like AI truly had thoughts. But in April 2026, a team of 12 researchers including Usha Bhalla, Thomas Fel, and Can Rager published a sobering paper suggesting we may have fundamentally misunderstood how AI represents concepts. If concepts are continuous manifolds rather than discrete directions, then SAEs—the much-hyped "AI microscope"—may be showing us the wrong picture.

Opening the black box: a decade of mechanistic interpretability

Since AlexNet's dominance in 2012, deep learning models have worked brilliantly without anyone knowing why. Early interpretability efforts centered on feature visualization—using gradient ascent to generate images maximally activating individual neurons (Chris Olah's team at Google Brain). But this revealed a fatal flaw: single neurons are often polysemantic, responding to academic citations, English dialogue, HTTP requests, and Korean text all at once.

In 2017, the famous word2vec analogy—king minus man plus woman yields queen—inspired the Linear Representation Hypothesis: concepts are represented as directions in activation space, and a concept's strength is measured by the projection of the activation vector onto that direction. If true, understanding neural networks becomes a geometry problem: find all concept directions and you have AI's vocabulary.

Superposition: cheating magic in high dimensions

The hypothesis faces a mathematical obstacle: a layer with n = 100,000 neurons can hold only 100,000 orthogonal directions, yet GPT-4-class models clearly "know" millions or billions of concepts.

In 2022, Anthropic's Nelson Elhage, Chris Olah, et al. revealed the answer in *Toy Models of Superposition*: neural networks cheat in high-dimensional space. While only n exactly orthogonal vectors exist, the Johnson-Lindenstrauss lemma shows you can pack roughly exp(n) nearly-orthogonal vectors, with negligible interference between them. Since real-world concepts are sparse (a text rarely mentions "quantum entanglement" and "French onion soup" together), this interference rarely causes problems.

Superposition explains polysemanticity: each real neuron is a linear combination of many features. It also suggested a fix—recover the hidden "ideal neurons" and restore AI's true vocabulary.

Sparse autoencoders: AI's Rosetta Stone?

SAEs are autoencoders whose hidden layer is far larger than the input (typically 4–32×), combined with strict sparsity: only a few hidden units may fire at any time. Trained to reconstruct a model's activations, SAEs were hoped to "de-superpose" entangled features into monosemantic units.

Anthropic validated SAEs on small transformers in 2023 and scaled them to Claude 3 Sonnet in 2024; OpenAI and Google DeepMind followed. But all of this rests on an implicit assumption: concepts correspond to independent linear directions. Is that assumption correct?

Concepts are manifolds, not directions

Bhalla et al. borrow a deep insight from cognitive science and neuroscience: many concepts are continuous manifolds, not discrete points. Color forms a ring-like manifold; emotion is describable as a two-dimensional manifold of valence and arousal.

In 2022, a Stanford team led by Surya Ganguli (*Science Advances*) measured concept geometry in macaque inferior temporal (IT) cortex while monkeys viewed 64 concepts (bears, cars, etc.). Each concept formed a low-dimensional manifold—roughly a 35-dimensional ellipsoid embedded in a 2048-dimensional space—and semantic relationships mapped directly onto geometric distances between manifolds (bear–wolf close, bear–car far).

If brains represent concepts as manifolds, AI likely does too.

The "dilution" dilemma

Bhalla et al. developed a rigorous theoretical framework showing SAEs can capture manifolds in two equivalent ways:

• Global capture: a compact set of atomic features whose linear span contains the manifold (like approximating a circle with many lines).
• Local capture: the manifold distributed across many features, each covering a local patch (like tiling a sphere).

The unsettling finding: real-world SAEs do neither. They fall into an intermediate state called dilution—features attempt partial global coverage and partial local patching, but the manifold's structure smears across many features, with no clear pattern visible in any single one. This explains why SAE researchers often find individual features frustratingly vague.

The deeper implication: if concepts are manifolds, both existing SAE architectures and the entire direction-based interpretability paradigm may require fundamental restructuring. Future representation-learning methods should treat geometric objects—manifolds, subspaces, topological structures—as the basic units of interpretability.

From directions to manifolds: a paradigm shift

Three takeaways:

1. SAEs are not an all-purpose AI microscope. They are powerful, but built on specific mathematical assumptions; if those don't match AI's true representation structure, the picture is misleading—like using a particle microscope to observe waves. 2. Concepts are likely richer than the Linear Representation Hypothesis suggests. "Red" is a spectrum, not a binary switch; "anger" a continuous dimension, not a boolean. If AI also uses manifolds, its representations may be closer to human cognition than we thought—but we need new tools: rather than isolated feature directions, we should search for coherent groups of atoms, analogous to population coding in neuroscience. 3. Does AI really "understand"? If concepts were mere directions, understanding would reduce to geometric proximity. If they are manifolds, understanding means grasping manifold geometry—knowing how to navigate it, which changes are "small" and which are "fundamental." Echoing Henri Poincaré: "Geometry is the art of reasoning about a group of operations." Perhaps understanding a concept means mastering a group of operations on its manifold.

Conclusion

Bhalla et al. do not dismiss SAEs—they remain among the strongest interpretability tools available. But the lesson is: a tool is not truth. What we see through SAEs is filtered through their direction-oriented design; if AI's true conceptual structure is manifold-like, the image will always be distorted.

Does AI understand "dog"? At the direction level, perhaps there is a "dog direction." At the manifold level, perhaps there is a "dog manifold," populated by continuous variation from chihuahua to Tibetan mastiff. True understanding may not be standing at a point, but being able to roam freely across the manifold.

---

References

1. Bhalla, U., Fel, T., Rager, C. et al. *Do Sparse Autoencoders Capture Concept Manifolds?* arXiv:2604.28119 [cs.LG] (2026). 2. Elhage, N. et al. *Toy Models of Superposition.* Transformer Circuits Thread (2022). 3. Bricken, T. et al. *Towards Monosemanticity: Decomposing Language Models With Dictionary Learning.* Anthropic (2023). 4. Templeton, A. et al. *Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet.* Anthropic (2024). 5. Gao, L. et al. *Scaling and Evaluating Sparse Autoencoders.* OpenAI (2024). 6. Chung, Y. et al. *Scaling in Representation Learning.* *Nature Communications* 14, 5108 (2023). 7. Abdelnour, F., Voss, H.U. & Raj, A. *From sensory to perceptual manifolds: The twist of neural geometry.* *Science Advances* 11, eadv0431 (2025). 8. Chung, S. et al. *Neural representational geometry underlies few-shot concept learning.* *PNAS* 119, e2205650119 (2022). 9. Mikolov, T., Yih, W. & Zweig, G. *Linguistic Regularities in Continuous Space Word Representations.* *NAACL-HLT* (2013). 10. Olah, C. et al. *Feature Visualization.* *Distill* (2017).