The Answer-Basin Hypothesis: Your Linear Probe Detects Answer-Distribution Statistics, Not Concepts

Forum topic · ✨步子哥 · 2026-09-22

Summary

A Chinese tech forum post discusses the Answer-Basin Representation Hypothesis, presented in the paper "We Are Not Probing or Steering Concepts" (arXiv:2609.24821) from University of Queensland, MBZUAI, and Tokyo Institute of Science. The central claim: linear probes and steering vectors applied to LLM hidden states do not encode concepts like "correctness" or "safety"; instead they encode statistics of the answer measure — the probability distribution over candidate answers derived from the model's generation distribution. Three key experiments support this: (1) pre-generation probes capture answer concentration while post-generation probes capture relative basin mass; (2) flipping the label-to-basin-mass relationship in training data causes probe predictions to invert on unseen questions; (3) probes trained only on pairs of incorrect answers still recover basin-mass rankings. Steering experiments show that reversed training data yields nearly opposite steering directions under shared PCA projection, and that a fixed steering vector produces larger gains for answers with higher initial basin mass — meaning its conceptual effect can reverse across questions. The author frames this as a clean instance of the proxy-target trap: probe accuracy aligns with concept detection on the training distribution but decouples out of distribution, challenging interpretability findings and safety-steering reliability.

The Answer-Basin Hypothesis: You Think You're Probing Concepts, But You're Only Probing Statistics of the Answer Distribution

*Structured summary of a Chinese forum post reviewing the paper "The Answer-Basin Representation Hypothesis: We Are Not Probing or Steering Concepts" (arXiv:2609.24821, code: github.com/V1centNevwake/answer-basin-representation; University of Queensland + MBZUAI + Tokyo Institute of Science).*

The unsettling experiment

A linear probe trained to read "did the model answer correctly" from hidden states works well on SafetyBench. But when the label-to-answer-quality relationship is reversed in the training data (correct answers labeled wrong and vice versa, with identical training objectives), the probe's predictions invert overall on unseen questions. The probe is not detecting a "concept" — it is detecting a statistic of the answer distribution, which merely happened to align with the concept labels.

Key concepts

Answer basin: the set of all generations leading to the same answer (e.g., every continuation of "2+3=" that yields "5").
Basin mass: the total probability of all continuations pointing to one answer — how confident the model is in it.
Answer measure: the distribution over answers obtained as the pushforward of the generation distribution.

Core observation

Concept-related linear structure in hidden states emerges from differences in the answer measure. The direction separating "model picks A" from "model picks B" looks like it encodes "correct vs. incorrect" but actually encodes "high basin mass vs. low basin mass". These usually align, but are not the same thing.

Key points from the three experiments

Experiment 1: Pre-generation, linear directions encode "answer concentration" (preference strength); post-generation, they encode "relative basin mass". Both are statistics of the answer measure, not concepts.
Experiment 2: Reversing the label–basin-mass relationship (correct answers having *lower* basin mass) produces probes whose predictions invert on unseen questions — same label rules, same cross-entropy loss, yet the "concept" flips.
Experiment 3: A probe trained only to compare basin mass between two incorrect answers still recovers basin-mass rankings on new questions. If it truly detected correctness, it could not learn anything from incorrect-only comparisons.

Steering experiments

Steering vectors trained on data with reversed correctness–basin-mass relationships end up nearly antipodal to original vectors under a shared PCA projection, despite unchanged labels.
A fixed steering vector gives larger relative gains to answers with higher initial basin mass — the same vector can push toward "correct" on one question and "incorrect" on another. The effect depends on the interaction between vector direction and the question's initial basin-mass distribution.

Implications

Interpretability: Many probe-based claims of "concept X exists linearly in the model" need re-examination. The linear representation hypothesis is not disproven — but the structure originates from the answer distribution's geometry, not concept labels.
Safety steering: A steering vector that works in-distribution may invert out-of-distribution when basin-mass distributions differ.
Evaluation: This is a clean instance of the proxy-target trap — probe accuracy is a proxy aligned with concept detection on the training distribution but decoupled elsewhere.

Connecting to the "proxy target trap"

The post situates the paper alongside similar traps: RAG retrieval relevance ≠ trustworthiness; moral label aggregation ≠ ground truth; V-JEPA pixel reconstruction ≠ rotation understanding; probe accuracy ≠ concept detection. The paper's contribution is turning the trap from a philosophical warning into a computable diagnosis: for any probe, ask which answer-measure statistic it detects and whether that statistic aligns with your concept labels on the target distribution.

Limits acknowledged

The hypothesis targets concept-related linear structure (correctness, safety, social bias), not necessarily syntactic features or positional encodings.
Experiments mainly cover Qwen and Gemma models; broader scaling validation is needed.
The deeper question — whether "concepts" are mere byproducts of the answer distribution or have independent ontological status expressed through it — remains open.

Conclusion

The title "We Are Not Probing or Steering Concepts" refines rather than refutes the linear representation hypothesis: linear structure exists, but it arises from the statistical structure of the answer distribution, and its alignment with concept labels is fragile, reversible, and possibly deceptive. Interpretability researchers should run label–basin reversal experiments; safety practitioners must evaluate steering stability under changing basin-mass distributions, not just in-distribution performance.

---

Paper: The Answer-Basin Representation Hypothesis: We Are Not Probing or Steering Concepts arXiv: https://arxiv.org/abs/2609.24821 Code: https://github.com/V1centNevwake/answer-basin-representation Affiliations: University of Queensland + MBZUAI + Tokyo Institute of Science