A Midlife Crisis for Mechanistic Interpretability: 30 Leading Researchers Say the Field Is Heading in the Wrong Direction
> Core claim: Mechanistic Interpretability (MI) is going through an identity crisis. It has spent five years tracing circuits in GPT-2-sized models while remaining helpless against the emergent behaviors of large language models. In early 2025, 30+ researchers from Anthropic, Redwood, Mila, and other institutions jointly released what amounts to a confession — not a celebration of achievements, but a list of open problems. If MI cannot solve at least half of them within three years, AI safety will rest on a pile of unverifiable assumptions.
---
1. An Awkward Fact: The Better We Understand Small Models, the Less We Understand Large Ones
In 2020, OpenAI released GPT-3. That same year, Chris Olah's team began using circuit tracing to analyze GPT-2. Five years later:
| Year | MI's focus models | Parameters | Frontier model | Parameters | Gap | |:---:|:---:|:---:|:---:|:---:|:---:| | 2020 | GPT-2 | 1.5B | GPT-3 | 175B | 117x | | 2022 | GPT-2 / small Transformers | < 1B | GPT-4 | ~1T | 1000x+ | | 2025 | Still GPT-2 scale | < 1B | GPT-4o / o3 | unknown | ∞ |
> Problem: MI has spent five years "practicing" on models under 1B parameters, while industry has deployed models that may exceed a trillion parameters. It is like a biologist spending five years studying the fruit fly nervous system, then being asked to explain the human brain — with the report due tomorrow.
The Sharkey et al. (2025) review does not shy away from this awkwardness. Instead, it lists it as the first-priority open problem:
> "Our methods require both conceptual and practical improvements to reveal deeper insights."
In plain language: our methods are not good enough.
---
2. Open Problem One: Superposition — Neural Networks' High-Dimensional Magic
This is the most fundamental and thorniest problem in MI.
2.1 The Problem
A neural network layer has \(d\) neurons, but the number of concepts it needs to represent may far exceed \(d\). What to do?
> Answer: Superposition. The model "stacks" multiple concepts on the same neuron or direction, distinguishing them through sparse activation.
> Superposition: Proposed by Elhage et al. (2022) at Anthropic, it refers to neural networks representing far more concepts than dimensions by encoding multiple features as sparse activations along non-orthogonal directions. It resembles the superposition of states in quantum mechanics — you cannot directly observe a single concept, only "excite" it via specific inputs.
2.2 Why It Matters
If superposition is ubiquitous, then:
| Traditional MI method | Assumption | Reality under superposition | |:---|:---|:---| | Finding one-to-one "direction-concept" mappings | Each direction represents one concept | ❌ One direction represents a combination of concepts | | Activating a single neuron to observe outputs | Neurons are interpretable | ❌ Single-neuron activations lack clear semantics | | Intervening on a single feature | Features are independent | ❌ Features are highly entangled |
> Conclusion: Superposition means the traditional "find one direction = understand one concept" methodology may simply not work on large models.
---
3. Open Problem Two: Causality vs Correlation — You Found a Circuit, Now What?
This is MI's most dangerous illusion.
3.1 The Circuit-Tracing Trap
Suppose intervention experiments show that in GPT-2, the "indirect object identification (IOI)" task relies on a specific circuit. You proudly announce: "I understand this circuit!"
But Sharkey et al. raise a pointed question:
> "Does identifying a circuit mean we understand the mechanism, or just the correlation?"
| Level | Question | Difficulty | |:---|:---|:---:| | Level 1: Correlation | "This neuron activates when cats are mentioned" | ⭐ | | Level 2: Circuit | "This circuit participates in the IOI task" | ⭐⭐ | | Level 3: Causation | "Removing this circuit causes IOI to fail" | ⭐⭐⭐ | | Level 4: Mechanism | "Why does this circuit work this way" | ⭐⭐⭐⭐⭐ |
> MI mostly sits at Levels 2–3. Real "understanding" requires Level 4.
3.2 The Limits of Intervention
Current MI intervention methods (like activation patching) carry a hidden assumption:
> Assumption: local interventions can verify global mechanisms.
But this assumption may be wrong. Distributed representations in large models mean:
- A behavior may be implemented by multiple overlapping circuits
- Intervening on one circuit, others may "compensate"
- The causal effect you observe may just be "the bottleneck easiest to intervene on," not the real mechanism
- Do circuits found in small models still exist in large models?
- Do large models use entirely different representational strategies?
- Do emergent capabilities correspond to emergent circuits, or completely different mechanisms?
- Interpretability ≠ controllability
- Understanding a mechanism ≠ being able to modify it without side effects
- Local understanding ≠ global understanding
- "Circuit-tracing fundamentalists" — who believe that with enough time, existing methods can understand any model.
- Blind believers in "interpretability is safety" — who treat correlation as causation and local understanding as global control.
- Researchers ignoring socio-technical challenges — MI is not purely a math problem; its applications have real ethical consequences.
> Activation Patching: A causal inference technique that replaces activations at some layer of the model with corresponding activations from another run (usually after modifying the input), observing output changes. If replacement changes behavior, the layer/position is considered involved in that behavior. But this assumes modularity and local causality — assumptions that may not hold in highly distributed large models.
---
4. Open Problem Three: Scalability — A Methodological Cliff
This is MI's most practical challenge.
4.1 Scalability Bottlenecks of Current Methods
| MI method | Time complexity | Largest applicable model | |:---|:---:|:---:| | Manual circuit tracing | \(O(d^2)\) per layer | ~100M | | Automatic circuit discovery (ACDC) | \(O(d^3)\) | ~1B | | Sparse autoencoders (SAE) | \(O(d \cdot k)\) | ~10B | | What is needed | \(O(d \cdot \log d)\) | 1T+ |
> Current status: SAEs are the most promising route to 10B+, but remain computationally expensive for 100B+ models.
4.2 The Blind Spot in Scaling Laws
A neglected question: do the regularities MI discovers change with model scale?
There are almost no systematic answers yet.
---
5. Open Problem Four: Socio-Technical Challenges — MI's Ethical Dilemma
Possibly the most neglected problem in the field.
5.1 MI's Dual Use
| Application | Value | Risk | |:---|:---|:---| | AI safety auditing | Identifying harmful capabilities | Equally usable to evade safety detection | | Capability enhancement | Optimizing performance | May accelerate development of dangerous capabilities | | Transparency | Increasing public trust | May be used to "whitewash" untrustworthy systems |
> Core tension: MI's tools and methods are value-neutral, but their applications have clear ethical directionality. Have researchers considered the potential for misuse when publishing papers?
5.2 The "Interpretability Is Safety" Illusion
The MI field carries an implicit narrative:
> "If we can explain the model, we can control the model."
But Sharkey et al. point out this is an unproven assumption:
> A bet: "Interpretability is safety" is one of AI safety's biggest cognitive traps. Understanding how a bomb explodes does not mean you can safely defuse it — especially with only 30% understanding.
---
6. My Bets
I bet $1,000: by the end of 2027, mechanistic interpretability will undergo a fundamental methodological shift — from "circuit-tracing fundamentalism" to "dynamic behavioral understanding," with SAEs and related distributed analysis methods becoming mainstream.
Why?
1. Circuit tracing hits a ceiling on large models: GPT-2-level analysis cannot be directly extrapolated to GPT-4 scale. This is structural, not engineering. 2. SAE breakthroughs: Anthropic's 2024 SAE work showed we can extract interpretable feature directions without layer-by-layer tracing. This is the only method with hope of scaling to 100B+. 3. Emergence requires new tools: Large models' emergent behaviors (in-context learning, chain-of-thought reasoning) may not correspond to any fixed circuit. Understanding them requires dynamic analysis tools, not static circuit diagrams. 4. Industry demand: As AI regulation tightens (EU AI Act, US executive orders), practical demands for interpretability will force methodological innovation.
Who is the enemy?
7. Why This Review Matters
This is not an ordinary literature review. It is a collective reflection by ~30 of the field's leading researchers.
Its key messages: 1. MI has made progress, but far from enough 2. Current methodologies have fundamental limitations 3. Without solving these open problems, AI safety will rest on unstable foundations 4. The field needs new conceptual frameworks, new tools, and new evaluation standards
In an era of exponentially growing AI capabilities, MI's "open problems list" is like a diagnostic report — telling us what is still missing from our toolbox as we try to understand these increasingly powerful systems.
> Final word: Mechanistic interpretability may be one of AI safety's most important subfields. But Sharkey et al.'s review reminds us: important does not mean mature. The field needs bolder hypotheses, more interdisciplinary borrowing, and more challenges to its basic methodology. There is no time to waste.
---
Paper Details
| Item | Content | |:---|:---| | Title | Open Problems in Mechanistic Interpretability | | Authors | Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, Tom McGrath | | Institutions | Anthropic, Redwood Research, Mila, MIT, Harvard, and others | | arXiv ID | 2501.16496 | | Date | 2025-01-27 | | Core contribution | Systematic mapping of conceptual, practical, and socio-technical open problems in mechanistic interpretability | | Key topics | Superposition, Causality, Scalability, Socio-Technical Challenges, Methodological Improvements |