The Answer-Basin Hypothesis: You're Not Probing "Concepts"—You're Probing Statistics of the Answer Distribution
A Disturbing Experiment
Suppose you train a linear probe that reads "did the model answer correctly or incorrectly" out of a language model's hidden states. It works well for safety detection on SafetyBench, so you conclude the model contains a linear direction encoding a "safety" concept.
Then you run a small experiment: flip the labels of "correct answer" and "wrong answer" in the training data—what was correct is now labeled wrong, and vice versa. The training objective and loss function stay the same; only the relation between labels and answer quality is reversed.
Result: the probe's predictions on unseen questions flip wholesale.
What does this mean? The probe was never detecting a "concept"—it was detecting some statistic of the answer distribution. When that statistic happens to align with your concept labels, the probe looks like it detects concepts; when the alignment reverses, the probe's "concept" flips too.
This is the core finding of the Answer-Basin Representation Hypothesis, from a paper whose title is blunt to the point of provocation: "We Are Not Probing or Steering Concepts."
Answer Basins: Starting from the Generation Distribution
To understand the hypothesis, you first need "answer basin."
Consider the math question "2+3=?". The model might generate "2+3=5", "2+3=5, because 5 is the sum of 2 and 3", and so on. All these generation paths point to the same answer, "5". The set of all continuations producing "5" forms the basin of that answer—like water draining into the same depression.
A basin has mass: the total probability of all continuations pointing to that answer. The more confident the model is in "5", the larger its basin mass.
Listing the basin masses of all possible answers yields an answer measure—a probability distribution over the answer space, pushed forward from the model's generation distribution. It fully describes the model's answer distribution for the question.
Key Observation: Linear Structure Follows the Answer Measure
The paper's central observation: concept-related linear structure emerges from differences in the answer measure.
Concretely, for a question with candidate answers A and B, where A's basin mass far exceeds B's, "model picks A" and "model picks B" states separate along some direction in hidden-state space. That direction looks like it encodes "correct vs. wrong"—but the paper shows it actually encodes "high basin mass vs. low basin mass".
Why does this distinction matter? Because "correct vs. wrong" and "high vs. low basin mass" are aligned in most cases—models are usually more confident in correct answers. But they are not the same thing. When they diverge, the probe's behavior exposes its true nature.
The paper runs three key experiments:
Experiment 1: Probes capture answer concentration pre-generation, relative basin mass post-generation
Before the model generates an answer, hidden-state linear directions encode "answer concentration"—how strongly the model prefers a particular answer. After generation, they encode "relative basin mass"—the contrast between answers' masses.
Neither is the "concept" itself; both are statistics of the answer measure.
Experiment 2: Flipping the label–basin relationship flips probe predictions
The most unsettling experiment. The researchers constructed training data where the basin-mass relation between correct and wrong answers is reversed—the correct answer has *smaller* basin mass. Probes trained on this data produce predictions opposite to the original probe on unseen questions.
The labeling rule didn't change ("correct" stayed "correct"), nor did the training objective (cross-entropy). But reversing the label–basin-mass relation flipped the probe's "concept". This shows the probe never learned "correctness"—it learned "basin-mass level", which merely happens to align with correctness.
Experiment 3: Training on only two wrong answers still recovers basin-mass ordering
Even more subtle. During training, the probe only sees labels for "which of two wrong answers has larger basin mass"—correct answers never appear. Yet on unseen questions, this probe recovers the basin-mass ordering of candidate answers.
If the probe truly detected "correctness", it couldn't learn anything about "correct" from comparing two wrong answers. But it did—because what it detects was never correctness; it was basin mass.
Steering: Reverse the Directions, Reverse the Effects
Probing reads; steering writes. The paper ran steering experiments too.
Reversing the basin-mass relation for correct/wrong answers in the training data and training steering vectors on it yields vectors that, in a shared PCA projection, point nearly opposite to the original ones—even though the labels ("this is the correct answer") never changed.
More subtly: when a fixed steering vector is applied to different questions, answers with larger initial basin mass gain more relative boost. The same steering vector can produce opposite concept-level effects—pushing "correct" on question A but "wrong" on question B.
This is a deep result: a steering vector's effect isn't determined by its direction alone, but by the interaction between direction and the question's initial basin mass. You think you're steering "correctness"; what you're actually doing is "amplifying high-basin-mass answers"—whether that answer is correct depends on the initial distribution.
What This Means: Three Levels of Impact
1. Interpretability
Linear probes are a cornerstone tool of interpretability research. If probes detect statistics rather than concepts, many conclusions of the form "the model contains a linear representation of concept X" need re-examination.
The paper doesn't say the linear representation hypothesis is wrong—linear structure does exist. But it identifies the structure's origin: not "the model learned concepts", but "the model's answer distribution shaped the geometry of its hidden states". Concept labels merely happen to align with that structure.
2. Safety steering
If steering vectors manipulate statistics of the answer distribution rather than "concepts", then the reliability of safety steering needs reevaluation. A safety vector that works on the test set may produce reversed effects out of distribution—because new questions' basin-mass distributions may differ from the training set's.
3. Evaluation methodology
The finding echoes a broader pattern: proxy failure ≠ real target intact. Probe accuracy is a proxy; it aligns with "concept detection" on the training distribution but can decouple entirely out of distribution. The paper provides a clean demonstration: the same probe, after label–basin reversal, may keep the same "accuracy" while detecting something entirely different.
Connection to the "Proxy Target Trap"
This paper is one of the cleanest instances of the "proxy target trap" I've seen recently:
- RAG retrieval relevance ≠ trustworthiness: a retrieved document being relevant doesn't make it reliable.
- Aggregated moral labels ≠ ground truth: aggregating moral annotations can diverge from genuine moral judgment.
- V-JEPA pixel reconstruction ≠ rotation perception: reconstructing pixels doesn't mean understanding rotation.
- Probe accuracy ≠ concept detection: distinguishing right from wrong doesn't mean detecting "correctness".
The Answer-Basin hypothesis's contribution: it doesn't just flag the trap; it gives a precise account of what the proxy actually measures—statistics of the answer measure. This turns the "proxy target trap" from a philosophical warning into a computable diagnosis: given a probe, ask which answer-measure statistic it detects, and whether that statistic aligns with your concept labels on the target distribution.
An Honest Assessment: The Hypothesis's Boundaries
The paper doesn't claim the hypothesis explains all linear structure. It focuses on "concept-related linear structure"—correctness, safety, social bias, i.e., concepts tightly tied to the model's output distribution. For other kinds of linear structure (syntax features, positional encodings), the answer-basin hypothesis may not apply.
Also, the experiments were run mainly on Qwen and Gemma; coverage of model scale and type is limited. Whether the hypothesis holds across all models at all scales needs more validation.
A deeper question: what exactly is the relation between "answer-measure statistics" and "concepts"? The paper shows the former gives rise to the latter, but does that mean "concepts" are merely byproducts of the answer distribution? Or do concepts have independent ontological status, merely manifesting in hidden states through the answer distribution? The paper doesn't dig into this, but it may be the crux of follow-up work.
Conclusion
"We Are Not Probing or Steering Concepts" doesn't negate the linear representation hypothesis—it refines it: linear structure exists, but it originates from the statistical structure of the answer distribution, not from concept labels themselves. Concept labels merely align with that structure on the training distribution—an alignment that is fragile, reversible, and gameable.
For interpretability researchers, this is a challenge you must face: what does your probe actually detect—a concept, or an answer-measure statistic? To answer, run the label–basin reversal experiment: if the probe's predictions flip too, you weren't detecting a concept.
For safety-steering practitioners, it's a warning: steering effects depend on the initial basin-mass distribution. Steering that works in-distribution may reverse out of distribution. When evaluating steering methods, don't just measure in-distribution performance—test stability under shifts in the basin-mass distribution.
This is a paper that makes you rethink "linear representations". The linear structure is still there—but the story behind it is far more complicated than we assumed.
---
Paper: The Answer-Basin Representation Hypothesis: We Are Not Probing or Steering Concepts arXiv: https://arxiv.org/abs/2609.24821 Code: https://github.com/V1centNevwake/answer-basin-representation Institutions: University of Queensland + MBZUAI + Institute of Science Tokyo