Midlife Crisis in Mechanistic Interpretability: 30 Top Researchers Say the Field May Be Heading in the Wrong Direction
> Core claim: Mechanistic Interpretability (MI) is going through an identity crisis. It spent five years tracing circuits in GPT-2-sized models while remaining helpless against the emergent behaviors of large language models. In early 2025, 30+ researchers from Anthropic, Redwood Research, Mila, and other institutions jointly published a paper — not a celebration of achievements, but a list of open problems. If MI cannot solve at least half of them within three years, AI safety will rest on a pile of unverifiable assumptions.
---
1. An Awkward Fact: The Better We Understand Small Models, the Less We Understand Large Ones
In 2020, OpenAI released GPT-3. That same year, Chris Olah's team began analyzing GPT-2 with circuit tracing. Five years later:
| Year | MI's Focus Model | Params | Frontier Model | Params | Gap | |:---:|:---:|:---:|:---:|:---:|:---:| | 2020 | GPT-2 | 1.5B | GPT-3 | 175B | 117x | | 2022 | GPT-2 / small Transformers | < 1B | GPT-4 | ~1T | 1000x+ | | 2025 | Still GPT-2 level | < 1B | GPT-4o / o3 | Unknown | ∞ |
> Problem: MI has spent five years "practicing" on sub-1B models while industry deploys models that may exceed a trillion parameters. It is like a biologist spending five years on the fly nervous system and then being asked to explain the human brain — report due tomorrow.
The Sharkey et al. (2025) review does not avoid this awkwardness. Instead, it lists it as a first-priority open problem:
> "Our methods require both conceptual and practical improvements to reveal deeper insights."
In plain words: our methods are not good enough.
---
2. Open Problem #1: Superposition — The High-Dimensional Magic of Neural Networks
The most fundamental and thorny problem in MI.
2.1 The Problem
A neural network layer has \(d\) neurons, but the number of concepts it needs to represent may far exceed \(d\). The answer: Superposition — the model "stacks" multiple concepts on the same neuron or direction, distinguishing them through sparse activation.
> Superposition: Proposed by Elhage et al. (2022) at Anthropic — neural networks, when dimension-limited, represent far more concepts than dimensions by encoding multiple features as sparse activations along non-orthogonal directions. Analogous to quantum superposition: you cannot directly observe a single concept, only "excite" it with specific inputs.
2.2 Why It Matters
If superposition is pervasive:
| Traditional MI Method | Assumption | Reality Under Superposition | |:---|:---|:---| | One-direction-one-concept mapping | Each direction = one concept | ❌ One direction = a combination of many concepts | | Observing single-neuron activations | Neurons are interpretable | ❌ Single-neuron activations have no clear semantics | | Intervening on single features | Features are independent | ❌ Features are highly entangled |
> Conclusion: Superposition means the traditional "find a direction = understand a concept" methodology may fundamentally fail on large models.
---
3. Open Problem #2: Causality vs Correlation — You Found the Circuit, Now What?
The most dangerous illusion in MI.
3.1 The Circuit-Tracing Trap
Suppose intervention experiments show that "indirect object identification" (IOI) in GPT-2 depends on a specific circuit, and you proudly announce: "I understand this circuit!" But Sharkey et al. pose a sharp question:
> "Does identifying a circuit mean we understand the mechanism, or just the correlation?"
| Level | Question | Difficulty | |:---|:---|:---:| | 1: Correlation | "This neuron activates on mentions of cats" | ⭐ | | 2: Circuit | "This circuit participates in the IOI task" | ⭐⭐ | | 3: Causal | "Removing this circuit breaks IOI" | ⭐⭐⭐ | | 4: Mechanism | "Why does this circuit work this way" | ⭐⭐⭐⭐⭐ |
> MI mostly sits at Levels 2–3. Real "understanding" requires Level 4.
3.2 The Limits of Intervention
Current intervention methods (e.g., activation patching) carry a hidden assumption:
> Assumption: local interventions can validate global mechanisms.
This may be wrong. Distributed representations in large models imply:
- One behavior may be implemented by multiple overlapping circuits
- Intervening on one circuit, others may "compensate"
- The causal effect you observe may just be "the bottleneck easiest to intervene on," not the true mechanism
- Do circuits found in small models still exist in large models?
- Do large models use entirely different representational strategies?
- Do emergent capabilities correspond to emergent circuits, or entirely different mechanisms?
- Interpretability ≠ controllability
- Understanding a mechanism ≠ modifying it without side effects
- Local understanding ≠ global understanding
- "Circuit-tracing fundamentalists" who believe enough time and existing methods can understand any model.
- Blind believers in "interpretability = safety" who mistake correlation for causation, local understanding for global control.
- Researchers ignoring socio-technical challenges — MI is not pure math; its applications have real ethical consequences.
> Activation Patching: A causal-inference technique that replaces activations at some layer of a run with corresponding activations from another run (usually with modified inputs) and observes output changes. If the swap changes behavior, the layer/position is deemed involved. But it assumes modularity and local causality — assumptions that may not hold in highly distributed large models.
---
4. Open Problem #3: Scalability — The Methodological Cliff
The most practical challenge facing MI.
4.1 Bottlenecks of Current Methods
| MI Method | Time Complexity | Largest Applicable Model | |:---|:---:|:---:| | Manual circuit tracing | \(O(d^2)\) per layer | ~100M | | Automatic Circuit Discovery (ACDC) | \(O(d^3)\) | ~1B | | Sparse Autoencoders (SAE) | \(O(d \cdot k)\) | ~10B | | What's needed | \(O(d \cdot \log d)\) | 1T+ |
> Status: SAEs are the most promising path to 10B+, but remain computationally expensive for 100B+ models.
4.2 The Blind Spot of Scaling Laws
A neglected question: do MI-discovered regularities change with model scale?
Almost no systematic answers exist.
---
5. Open Problem #4: Socio-Technical Challenges — MI's Ethical Dilemma
Possibly the most overlooked issue in MI.
5.1 Dual Use
| Application | Value | Risk | |:---|:---|:---| | AI safety auditing | Identifying harmful capabilities | Equally usable to evade safety detection | | Capability enhancement | Optimizing performance | May accelerate dangerous capabilities | | Transparency | Increasing public trust | May "whitewash" untrustworthy systems |
> Core tension: MI tools are value-neutral, but their applications carry clear ethical directionality. Do researchers consider misuse when publishing?
5.2 The "Interpretability = Safety" Illusion
The field carries an implicit narrative:
> "If we can explain the model, we can control the model."
But Sharkey et al. note this is an unproven assumption:
> Author's take: "Interpretability as safety" is one of the biggest cognitive traps in AI safety. Understanding how a bomb explodes doesn't mean you can defuse it safely — especially at 30% understanding.
---
6. The Author's Bet
I bet $1,000: by the end of 2027, MI will undergo a fundamental methodological shift — from "circuit-tracing fundamentalism" to "dynamic behavioral understanding," with SAEs and related distributed-analysis methods becoming mainstream.
Why?
1. Circuit tracing hits a ceiling on large models: GPT-2-level analysis cannot extrapolate to GPT-4-level. This is structural, not an engineering problem. 2. SAE breakthroughs: Anthropic's 2024 SAE work showed we can extract interpretable feature directions without layer-by-layer tracing — the only method promising to scale to 100B+. 3. Emergence demands new tools: Emergent behaviors (in-context learning, chain-of-thought reasoning) may not correspond to any fixed circuit. Understanding them requires dynamic analysis tools, not static circuit diagrams. 4. Industry demand: As AI regulation strengthens (EU AI Act, US executive orders), practical demands for interpretability will force methodological innovation.
Who is the enemy?
7. Why This Review Matters
This is not an ordinary literature review. It is a collective reflection by ~30 leading researchers. Its key messages:
1. MI has made progress, but far from enough 2. Current methodology has fundamental limitations 3. Without solving these open problems, AI safety rests on unstable foundations 4. The field needs new conceptual frameworks, new tools, new evaluation standards
In an era of exponential AI capability growth, MI's "open problems list" is a diagnostic report — it tells us what is missing from our toolbox as we try to understand increasingly powerful systems.
> Final word: Mechanistic interpretability may be one of the most important subfields of AI safety. But Sharkey et al.'s review reminds us: important does not mean mature. The field needs bolder hypotheses, more interdisciplinary borrowing, and more challenges to basic methodology. There is no time to waste.
---
Paper Details
| Item | Content | |:---|:---| | Title | Open Problems in Mechanistic Interpretability | | Authors | Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, Tom McGrath | | Institutions | Anthropic, Redwood Research, Mila, MIT, Harvard, and others | | arXiv ID | 2501.16496 | | Date | 2025-01-27 | | Core Contribution | Systematic survey of conceptual, practical, and socio-technical open problems in mechanistic interpretability | | Key Topics | Superposition, Causality, Scalability, Socio-Technical Challenges, Methodological Improvements |