Open Problems in Mechanistic Interpretability: A Survey of 30 Researchers on the Future of AI Explainability
In January 2025, more than 30 researchers from Anthropic, Redwood Research, Mila, MIT, and other institutions jointly released a forward-looking survey that systematically catalogues open problems in mechanistic interpretability (MI). Rather than celebrating achievements, the paper confronts deep challenges in conceptual foundations, methodology, and societal impact.
---
1. State of the Field and the Existing Gap
1.1 Five-Year Retrospective: From GPT-2 to GPT-4
Since MI emerged as an independent research area around 2020, notable local progress has been made:
| Milestone | Year | Contribution | Model scale | |:---|:---:|:---|:---:| | Olah et al. circuit visualization | 2020 | Introduced feature visualization and attribution | < 1B | | Elhage et al. mathematical framework | 2021 | Formalized Transformer circuits | < 1B | | IOI circuit discovery | 2022 | Located indirect-object-identification circuit in GPT-2 | 1.5B | | SAE feature extraction | 2023–2024 | Sparse autoencoders for interpretable features | 1B–10B | | Current frontier | 2025 | Scaling methods toward larger models | ~10B |
Key observation: Core MI methods (circuit tracing, feature visualization, SAEs) have largely been validated on models between 1B and 10B parameters, while deployed frontier models may exceed 100B–1T+ parameters. Sharkey et al. identify this methodology–deployment gap as the foremost issue.
1.2 The Survey's Structure
Sharkey et al. organize open problems into three layers:
| Layer | Focus | Representative questions | |:---|:---|:---| | Conceptual | Theoretical foundations | Generality of superposition; causation vs correlation | | Practical | Methodology and tools | Scalability, evaluation standards, automation | | Socio-technical | Ethics and applications | Dual-use risks, governance, transparency theater |
---
2. Conceptual Challenges: Understanding "Understanding"
2.1 The Generality of Superposition
Superposition (Elhage et al., 2022) describes how, when dimensional capacity is limited, neural networks encode multiple features along non-orthogonal directions with sparse activations. Formally, given an activation space $\mathbb{R}^d$ and a feature set $\{f_1, f_2, \dots, f_n\}$ with $n \gg d$, the model encodes features via an overcomplete, non-orthogonal basis $\{e_1, e_2, \dots, e_n\}$:
$$\text{activation} = \sum_{i=1}^{n} \alpha_i e_i, \quad \text{where } \alpha_i \text{ is sparse}$$
Open question: Is superposition universal across model scales and architectures? If so, how must traditional MI methods—which assume a one-to-one direction–feature correspondence—be fundamentally revised?
2.2 The Limits of Causal Inference
The dominant causal tool in MI is activation patching (also called interchange intervention):
1. Run the model on input $x$ and record activations $a_l(x)$ at layer $l$. 2. Run the model on a modified input $x'$ and record $a_l(x')$. 3. Replace $a_l(x)$ with $a_l(x')$ and observe output changes. 4. Infer layer $l$'s involvement if outputs change significantly.
Limitation: The method implicitly assumes local causality and modularity—that specific behaviors are realized by specific components. In highly distributed large models, behaviors may be realized by multiple overlapping circuits, and intervening on one component can trigger compensation elsewhere.
---
3. Practical Challenges: From Fruit Fly to Human Brain
3.1 Order-of-Magnitude Scalability Gap
| Method | Computational complexity | Largest current application | Target scale | |:---|:---:|:---:|:---:| | Manual circuit tracing | $O(d^2 \cdot L)$ | ~100M | 1T+ | | ACDC automated discovery | $O(d^3 \cdot L)$ | ~1B | 1T+ | | Sparse autoencoders (SAE) | $O(d \cdot k \cdot L)$ | ~10B | 1T+ | | Ideal method | $O(d \cdot \log d)$ | — | 1T+ |
Here $d$ is layer width, $L$ is layer count, and $k$ is the SAE expansion factor.
Key gap: Going from 10B to 1T+ parameters means a 100× scale increase. With current methods scaling as $d^2$ or $d^3$, this could translate to 10,000×–1,000,000× compute growth.
3.2 Methodological Blind Spots for Emergent Phenomena
Large language models exhibit emergent capabilities absent in smaller models:
- In-context learning
- Chain-of-thought reasoning
- Instruction following
- Tool use
- Path-integral methods that track information flow across multiple layers
- Phase-transition analysis that identifies critical points where model behavior qualitatively changes
- Multi-scale analysis spanning neurons, circuits, modules, and systems
Open question: Do these emergent capabilities correspond to identifiable circuits or features? Or are they inherently distributed dynamic processes that resist decomposition into static components?
Sharkey et al. stress that MI needs dynamic analysis tools to complement current static circuit methods.
---
4. Socio-Technical Challenges: The Ethical Dimension of Interpretability
4.1 The Dual-Use Dilemma
MI research outputs have clear dual-use characteristics:
| Application | Positive value | Potential misuse | |:---|:---|:---| | Safety auditing | Identify and mitigate harmful behaviors | Reverse-engineer ways to bypass safety mechanisms | | Capability enhancement | Optimize useful task performance | Optimize harmful task performance | | Transparency | Build public trust in AI systems | Provide an "interpretability veneer" for untrustworthy systems |
Open question: How should MI researchers evaluate and disclose dual-use risks? Should the field adopt biosafety-style "review and concern" mechanisms?
4.2 The Unproven Assumption "Interpretability Equals Safety"
The field operates under an implicit reasoning chain:
$$\text{Interpretable} \rightarrow \text{Predictable} \rightarrow \text{Controllable} \rightarrow \text{Safe}$$
Sharkey et al. note that each link contains an unproven assumption:
| Inference step | Implicit assumption | Counterexample | |:---|:---|:---| | Interpretable → Predictable | Understanding components suffices to predict system behavior | Emergent behaviors in complex systems cannot be derived from components | | Predictable → Controllable | Prediction entails intervention capability | Understanding cancer mechanisms ≠ curing cancer | | Controllable → Safe | Technical control guarantees safety | Nuclear power is controllable yet carries political/operational risks |
Conclusion: Interpretability is a necessary but not sufficient condition for safety. Treating MI as a silver bullet for AI safety is a dangerous oversimplification.
---
5. Frontier Directions: Possible Breakthrough Paths
5.1 Scaling Sparse Autoencoders (SAEs)
SAEs bypass traditional layer-by-layer circuit tracing by decomposing activations into sparse, interpretable feature directions:
$$\text{SAE}(x) = \text{Decoder}(\text{Encoder}(x)), \quad \|\text{Encoder}(x)\|_0 \ll d$$
Anthropic's 2024 work showed SAEs can extract interpretable features (e.g., "Golden Gate Bridge," "code syntax") without task-specific supervision.
Outlook: SAEs are the most promising route to 100B+ models, but training cost, feature stability, and feature interactions remain open problems.
5.2 Dynamic and Emergent Behavior Analysis
Static circuit analysis may be insufficient for understanding large-model dynamics. Emerging directions include:
6. Conclusion
*Open Problems in Mechanistic Interpretability* is not a pessimistic technical document but an honest self-assessment. Its core message: mechanistic interpretability is a young field facing significant challenges, and its current methodology is insufficient for increasingly capable AI systems.
Sharkey et al. set a clear research agenda: 1. Develop new theoretical frameworks that handle superposition and distributed representations. 2. Build tools that scale to 100B+ parameter models. 3. Establish rigorous causal-inference standards that go beyond correlational analysis. 4. Confront the socio-technical consequences of MI research head-on.
As AI capabilities continue to grow exponentially, the urgency of these open problems will only increase. Whether the MI field achieves breakthrough progress before critical thresholds may directly affect the foundations of AI safety.
---
Paper Details
| Item | Details | |:---|:---| | Title | Open Problems in Mechanistic Interpretability | | Authors | Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, Tom McGrath | | Institutions | Anthropic, Redwood Research, Mila, MIT, Harvard, University of Melbourne, et al. | | arXiv ID | 2501.16496 | | Date | 2025-01-27 | | Core contribution | Systematic catalogue of conceptual, practical, and socio-technical open problems in MI, with a proposed research agenda | | Key topics | Superposition, causality, scalability, socio-technical challenges, SAE, circuit tracing |