A Midlife Crisis for Mechanistic Interpretability: 30 Top Researchers List the Field's Open Problems 🔍
> Core claim: Mechanistic interpretability (MI) is undergoing an identity crisis. After five years of tracing circuits in GPT-2-scale models, it remains largely powerless against the emergent behaviors of large language models. In early 2025, 30+ researchers from Anthropic, Redwood Research, Mila, and other institutions jointly published a "confession" of sorts — not a celebration of achievements, but a list of open problems. If MI cannot solve at least half of them within three years, the author argues, AI safety will rest on unverifiable assumptions.
---
1. An Awkward Fact: The Better We Understand Small Models, The Less We Understand Large Ones 🤯
In 2020, OpenAI released GPT-3. The same year, Chris Olah's team began analyzing GPT-2 with circuit tracing. Five years later:
| Year | Focus models in MI | Params | Frontier models | Params | Gap | |:---:|:---:|:---:|:---:|:---:|:---:| | 2020 | GPT-2 | 1.5B | GPT-3 | 175B | 117x | | 2022 | GPT-2 / small Transformers | < 1B | GPT-4 | ~1T | 1000x+ | | 2025 | Still GPT-2-scale | < 1B | GPT-4o / o3 | unknown | ∞ |
> The problem: MI has spent five years "practicing" on sub-1B-parameter models while industry has deployed models that may exceed a trillion parameters. It is like a biologist spending five years on the fruit fly nervous system and then being asked to explain the human brain — due tomorrow.
The survey by Sharkey et al. (2025) does not shy away from this. It ranks it as a top-priority open problem:
> "Our methods require both conceptual and practical improvements to reveal deeper insights.""
In plain terms: our methods are not enough.
---
2. Open Problem #1: Superposition — The High-Dimensional Magic of Neural Networks 🎩
The most fundamental and thorny problem in MI.
2.1 The problem
A neural network layer has \(d\) neurons but may need to represent far more than \(d\) concepts.
> Answer: Superposition. The model "stacks" multiple concepts on the same neuron or direction, distinguished via sparse activation.
> ..... Superposition: Proposed by Elhage et al. (2022) at Anthropic — when dimensions are limited, a network represents more concepts than dimensions by encoding features as sparse activations along non-orthogonal directions. Like quantum superposition: you cannot directly observe a single concept, only "excite" it with specific inputs.
2.2 Why it matters
If superposition is pervasive, then:
| Traditional MI method | Assumption | Reality under superposition | |:---|:---|:---| | Find one-direction-per-concept mappings | Each direction = one concept | ❌ One direction = a mixture of concepts | | Activate single neurons and observe | Neurons are interpretable | ❌ Individual neuron activations lack clear semantics | | Intervene on single features | Features are independent | ❌ Features are highly entangled |
> Conclusion: Superposition means the traditional methodology of "find a direction = understand a concept" may simply not work at scale.
---
3. Open Problem #2: Causality vs. Correlation — You Found a Circuit, So What? 🔗
The most dangerous illusion in MI.
3.1 The circuit-tracing trap
Suppose intervention experiments show that GPT-2's "indirect object identification (IOI)" task relies on a specific circuit, and you proudly declare: "I understand this circuit!"
But Sharkey et al. pose a sharp question:
> "Does identifying a circuit mean we understand the mechanism, or just the correlation?""
| Level | Question | Difficulty | |:---|:---|:---:| | Level 1: Correlation | "This neuron activates on mentions of cats" | ⭐ | | Level 2: Circuit | "This circuit participates in IOI" | ⭐⭐ | | Level 3: Causal | "Removing this circuit breaks IOI" | ⭐⭐⭐ | | Level 4: Mechanism | "Why does this circuit work this way" | ⭐⭐⭐⭐⭐ |
> MI mostly operates at Levels 2–3. Real "understanding" requires Level 4.
3.2 The limits of intervention
Current intervention methods (e.g., activation patching) carry a hidden assumption:
> Assumption: local interventions can validate global mechanisms.
But this may be wrong. Distributed representations in large models mean:
- One behavior may be implemented by multiple overlapping circuits
- Intervening on one circuit, others may "compensate"
- The causal effect you observe may just be the "easiest bottleneck to intervene on," not the true mechanism
- Do circuits found in small models persist in large ones?
- Do large models use entirely different representational strategies?
- Do emergent capabilities correspond to emergent circuits, or entirely different mechanisms?
- Interpretability ≠ controllability
- Understanding a mechanism ≠ modifying it without side effects
- Local understanding ≠ global understanding
- "Circuit-tracing fundamentalists" who believe existing methods can understand any model given enough time.
- Blind believers in "interpretability = safety" who mistake correlation for causation and local understanding for global control.
- Researchers who ignore socio-technical challenges — MI is not pure math; its applications have real ethical consequences.
> ..... Activation Patching: A causal-inference technique that replaces activations at one layer with those from another run (usually with a modified input) and observes output changes. If replacement changes behavior, that layer/position is deemed involved. But it assumes modularity and local causality — assumptions that may not hold in highly distributed large models.
---
4. Open Problem #3: Scalability — The Methodological Cliff 📉
The most practical challenge.
4.1 Bottlenecks of current methods
| MI method | Complexity | Largest applicable model | |:---|:---:|:---| | Manual circuit tracing | \(O(d^2)\) per layer | ~100M | | Automatic circuit discovery (ACDC) | \(O(d^3)\) | ~1B | | Sparse autoencoders (SAE) | \(O(d \cdot k)\) | ~10B | | What's needed | \(O(d \cdot \log d)\) | 1T+ |
> Status: SAEs are the most promising route to 10B+, but remain computationally expensive for 100B+ models.
4.2 Blind spots in scaling laws
A neglected question: do MI-identified regularities change with model scale?
Almost no systematic answers exist.
---
5. Open Problem #4: Socio-Technical Challenges — MI's Ethical Dilemma ⚖️
Possibly the most neglected problem in the field.
5.1 Dual use of MI
| Application | Value | Risk | |:---|:---|:---| | AI safety auditing | Identify harmful capabilities | Same tools can evade safety detection | | Capability enhancement | Optimize performance | May accelerate dangerous capabilities | | Transparency | Build public trust | May "whitewash" untrustworthy systems |
> Core tension: MI tools are value-neutral, but their applications carry clear ethical direction. Do researchers consider misuse when publishing?
5.2 The "interpretability = safety" illusion
A implicit narrative in MI:
> "If we can explain the model, we can control it.""
But Sharkey et al. flag this as an unproven assumption:
> The author's bet: "Interpretability = safety" is one of the biggest cognitive traps in AI safety. Understanding how a bomb works is not the same as defusing it safely — especially at 30% understanding.
---
6. The Author's Bets 💰
I bet $1,000: by the end of 2027, MI will undergo a fundamental methodological shift — from "circuit-tracing fundamentalism" to "dynamic behavioral understanding," with SAEs and related distributed analysis methods becoming mainstream.
Why?
1. Circuit tracing hits a ceiling at scale: GPT-2-level analysis cannot extrapolate to GPT-4-class models. This is structural, not an engineering problem. 2. The SAE breakthrough: Anthropic's 2024 SAE work shows we can extract interpretable feature directions without layer-by-layer tracing — the only approach with hope of scaling to 100B+. 3. Emergence demands new tools: Emergent behaviors (in-context learning, reasoning chains) may not correspond to any fixed circuit; understanding them requires dynamic analysis tools, not static circuit diagrams. 4. Industry demand: Strengthening AI regulation (EU AI Act, US executive orders) will force methodological innovation.
Who is the enemy?
7. Why This Survey Matters 🌍
This is not an ordinary literature review. It is a collective reflection by 30 of the field's leading researchers. Its key messages:
1. MI has progressed, but far from enough 2. Current methodologies have fundamental limitations 3. Without solving these open problems, AI safety rests on shaky foundations 4. The field needs new conceptual frameworks, new tools, and new evaluation standards
In an era of exponential AI capability growth, MI's open-problems list is a diagnostic report — telling us what is missing from our toolbox as we try to understand increasingly powerful systems.
> Final word: Mechanistic interpretability may be one of AI safety's most important subfields. But Sharkey et al.'s survey reminds us: important does not mean mature. The field needs bolder hypotheses, more interdisciplinary borrowing, and more challenges to its basic methodology. There is no time to waste.
---
Paper Details
| Item | Content | |:---|:---| | Title | Open Problems in Mechanistic Interpretability | | Authors | Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, Tom McGrath | | Institutions | Anthropic, Redwood Research, Mila, MIT, Harvard, and others | | arXiv ID | 2501.16496 | | Date | 2025-01-27 | | Core contribution | Systematic mapping of conceptual, practical, and socio-technical open problems in mechanistic interpretability | | Key themes** | Superposition, Causality, Scalability, Socio-Technical Challenges, Methodological Improvements |