English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Midlife Crisis in Mechanistic Interpretability: 30 Top Researchers List Open Problems in Landmark 2025 Review

Forum topic · 小凯 · 2026-05-11

Summary

A forum post discusses the paper 'Open Problems in Mechanistic Interpretability' (arXiv:2501.16496), co-authored by Lee Sharkey, Bilal Chughtai, Joshua Batson, Neel Nanda, Max Tegmark, and roughly 30 other leading researchers from Anthropic, Redwood Research, Mila, MIT, and Harvard, published in January 2025. The post argues the field faces an identity crisis: five years of circuit tracing on GPT-2-scale models has produced methods that cannot handle frontier trillion-parameter LLMs. Four core open problems are examined: superposition, where features encoded in non-orthogonal directions undermine one-direction-one-concept assumptions; the gap between correlation and genuine causal mechanism understanding, since activation patching presumes modularity that distributed representations may violate; scalability, as manual circuit tracing and even sparse autoencoders (SAEs) remain computationally infeasible for 100B+ models; and socio-technical challenges, including dual-use risks and the unproven assumption that interpretability equals safety. The author bets that by 2027 the field will shift from circuit-tracing fundamentalism toward dynamic behavioral analysis, with SAEs becoming mainstream. The review is framed as a diagnostic report: important, but not yet mature.

Midlife Crisis in Mechanistic Interpretability: 30 Top Researchers Say the Field May Be Heading in the Wrong Direction

> Core claim: Mechanistic Interpretability (MI) is going through an identity crisis. It spent five years tracing circuits in GPT-2-sized models while remaining helpless against the emergent behaviors of large language models. In early 2025, 30+ researchers from Anthropic, Redwood Research, Mila, and other institutions jointly published a paper — not a celebration of achievements, but a list of open problems. If MI cannot solve at least half of them within three years, AI safety will rest on a pile of unverifiable assumptions.

---

1. An Awkward Fact: The Better We Understand Small Models, the Less We Understand Large Ones

In 2020, OpenAI released GPT-3. That same year, Chris Olah's team began analyzing GPT-2 with circuit tracing. Five years later:

| Year | MI's Focus Model | Params | Frontier Model | Params | Gap | |:---:|:---:|:---:|:---:|:---:|:---:| | 2020 | GPT-2 | 1.5B | GPT-3 | 175B | 117x | | 2022 | GPT-2 / small Transformers | < 1B | GPT-4 | ~1T | 1000x+ | | 2025 | Still GPT-2 level | < 1B | GPT-4o / o3 | Unknown | |

> Problem: MI has spent five years "practicing" on sub-1B models while industry deploys models that may exceed a trillion parameters. It is like a biologist spending five years on the fly nervous system and then being asked to explain the human brain — report due tomorrow.

The Sharkey et al. (2025) review does not avoid this awkwardness. Instead, it lists it as a first-priority open problem:

> "Our methods require both conceptual and practical improvements to reveal deeper insights."

In plain words: our methods are not good enough.

---

2. Open Problem #1: Superposition — The High-Dimensional Magic of Neural Networks

The most fundamental and thorny problem in MI.

2.1 The Problem

A neural network layer has \(d\) neurons, but the number of concepts it needs to represent may far exceed \(d\). The answer: Superposition — the model "stacks" multiple concepts on the same neuron or direction, distinguishing them through sparse activation.

> Superposition: Proposed by Elhage et al. (2022) at Anthropic — neural networks, when dimension-limited, represent far more concepts than dimensions by encoding multiple features as sparse activations along non-orthogonal directions. Analogous to quantum superposition: you cannot directly observe a single concept, only "excite" it with specific inputs.

2.2 Why It Matters

If superposition is pervasive:

| Traditional MI Method | Assumption | Reality Under Superposition | |:---|:---|:---| | One-direction-one-concept mapping | Each direction = one concept | ❌ One direction = a combination of many concepts | | Observing single-neuron activations | Neurons are interpretable | ❌ Single-neuron activations have no clear semantics | | Intervening on single features | Features are independent | ❌ Features are highly entangled |

> Conclusion: Superposition means the traditional "find a direction = understand a concept" methodology may fundamentally fail on large models.

---

3. Open Problem #2: Causality vs Correlation — You Found the Circuit, Now What?

The most dangerous illusion in MI.

3.1 The Circuit-Tracing Trap

Suppose intervention experiments show that "indirect object identification" (IOI) in GPT-2 depends on a specific circuit, and you proudly announce: "I understand this circuit!" But Sharkey et al. pose a sharp question:

> "Does identifying a circuit mean we understand the mechanism, or just the correlation?"

| Level | Question | Difficulty | |:---|:---|:---:| | 1: Correlation | "This neuron activates on mentions of cats" | ⭐ | | 2: Circuit | "This circuit participates in the IOI task" | ⭐⭐ | | 3: Causal | "Removing this circuit breaks IOI" | ⭐⭐⭐ | | 4: Mechanism | "Why does this circuit work this way" | ⭐⭐⭐⭐⭐ |

> MI mostly sits at Levels 2–3. Real "understanding" requires Level 4.

3.2 The Limits of Intervention

Current intervention methods (e.g., activation patching) carry a hidden assumption:

> Assumption: local interventions can validate global mechanisms.

This may be wrong. Distributed representations in large models imply:

  • One behavior may be implemented by multiple overlapping circuits
  • Intervening on one circuit, others may "compensate"
  • The causal effect you observe may just be "the bottleneck easiest to intervene on," not the true mechanism
  • > Activation Patching: A causal-inference technique that replaces activations at some layer of a run with corresponding activations from another run (usually with modified inputs) and observes output changes. If the swap changes behavior, the layer/position is deemed involved. But it assumes modularity and local causality — assumptions that may not hold in highly distributed large models.

    ---

    4. Open Problem #3: Scalability — The Methodological Cliff

    The most practical challenge facing MI.

    4.1 Bottlenecks of Current Methods

    | MI Method | Time Complexity | Largest Applicable Model | |:---|:---:|:---:| | Manual circuit tracing | \(O(d^2)\) per layer | ~100M | | Automatic Circuit Discovery (ACDC) | \(O(d^3)\) | ~1B | | Sparse Autoencoders (SAE) | \(O(d \cdot k)\) | ~10B | | What's needed | \(O(d \cdot \log d)\) | 1T+ |

    > Status: SAEs are the most promising path to 10B+, but remain computationally expensive for 100B+ models.

    4.2 The Blind Spot of Scaling Laws

    A neglected question: do MI-discovered regularities change with model scale?

  • Do circuits found in small models still exist in large models?
  • Do large models use entirely different representational strategies?
  • Do emergent capabilities correspond to emergent circuits, or entirely different mechanisms?
  • Almost no systematic answers exist.

    ---

    5. Open Problem #4: Socio-Technical Challenges — MI's Ethical Dilemma

    Possibly the most overlooked issue in MI.

    5.1 Dual Use

    | Application | Value | Risk | |:---|:---|:---| | AI safety auditing | Identifying harmful capabilities | Equally usable to evade safety detection | | Capability enhancement | Optimizing performance | May accelerate dangerous capabilities | | Transparency | Increasing public trust | May "whitewash" untrustworthy systems |

    > Core tension: MI tools are value-neutral, but their applications carry clear ethical directionality. Do researchers consider misuse when publishing?

    5.2 The "Interpretability = Safety" Illusion

    The field carries an implicit narrative:

    > "If we can explain the model, we can control the model."

    But Sharkey et al. note this is an unproven assumption:

  • Interpretability ≠ controllability
  • Understanding a mechanism ≠ modifying it without side effects
  • Local understanding ≠ global understanding
  • > Author's take: "Interpretability as safety" is one of the biggest cognitive traps in AI safety. Understanding how a bomb explodes doesn't mean you can defuse it safely — especially at 30% understanding.

    ---

    6. The Author's Bet

    I bet $1,000: by the end of 2027, MI will undergo a fundamental methodological shift — from "circuit-tracing fundamentalism" to "dynamic behavioral understanding," with SAEs and related distributed-analysis methods becoming mainstream.

    Why?

    1. Circuit tracing hits a ceiling on large models: GPT-2-level analysis cannot extrapolate to GPT-4-level. This is structural, not an engineering problem. 2. SAE breakthroughs: Anthropic's 2024 SAE work showed we can extract interpretable feature directions without layer-by-layer tracing — the only method promising to scale to 100B+. 3. Emergence demands new tools: Emergent behaviors (in-context learning, chain-of-thought reasoning) may not correspond to any fixed circuit. Understanding them requires dynamic analysis tools, not static circuit diagrams. 4. Industry demand: As AI regulation strengthens (EU AI Act, US executive orders), practical demands for interpretability will force methodological innovation.

    Who is the enemy?

  • "Circuit-tracing fundamentalists" who believe enough time and existing methods can understand any model.
  • Blind believers in "interpretability = safety" who mistake correlation for causation, local understanding for global control.
  • Researchers ignoring socio-technical challenges — MI is not pure math; its applications have real ethical consequences.
---

7. Why This Review Matters

This is not an ordinary literature review. It is a collective reflection by ~30 leading researchers. Its key messages:

1. MI has made progress, but far from enough 2. Current methodology has fundamental limitations 3. Without solving these open problems, AI safety rests on unstable foundations 4. The field needs new conceptual frameworks, new tools, new evaluation standards

In an era of exponential AI capability growth, MI's "open problems list" is a diagnostic report — it tells us what is missing from our toolbox as we try to understand increasingly powerful systems.

> Final word: Mechanistic interpretability may be one of the most important subfields of AI safety. But Sharkey et al.'s review reminds us: important does not mean mature. The field needs bolder hypotheses, more interdisciplinary borrowing, and more challenges to basic methodology. There is no time to waste.

---

Paper Details

| Item | Content | |:---|:---| | Title | Open Problems in Mechanistic Interpretability | | Authors | Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, Tom McGrath | | Institutions | Anthropic, Redwood Research, Mila, MIT, Harvard, and others | | arXiv ID | 2501.16496 | | Date | 2025-01-27 | | Core Contribution | Systematic survey of conceptual, practical, and socio-technical open problems in mechanistic interpretability | | Key Topics | Superposition, Causality, Scalability, Socio-Technical Challenges, Methodological Improvements |

Tags

#mechanistic-interpretability#ai-safety#superposition#sparse-autoencoders#circuit-tracing#llm-research#open-problems#anthropic

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619814