Linguistic Illegibility: When LLMs' Words Don't Match Their Internal Computation

*An in-depth Chinese forum commentary on* The Implications of Linguistic Illegibility for LLM Security *arXiv: 2609.02852 | Author: James Mickens*

Key points

• Core claim: An LLM's textual output — and even linguistic features recovered by mechanistic interpretability probes — does not faithfully reflect its internal computation. The author calls this gap "linguistic illegibility."
• Why it's inevitable: Real model computation happens in a high-dimensional activation space via matrix multiplications, attention weights, and nonlinear transforms. The mapping from this space to natural language is lossy, like compressing a symphony to MP3.
• Safety implications: Chain-of-thought monitoring, constitutional self-critique, and activation-probe-based deception detectors all implicitly assume linguistic outputs or human-labeled features are trustworthy — an assumption the paper dismantles.
• Proposed remedy: Taint tracking and sandboxing — treat all model output as untrusted data, enforce formal security invariants, and complement with robust virtualization, third-party auditing, and least-privilege design.
• Philosophical angle: The argument challenges the Turing test and echoes Wittgenstein's "even if a lion could speak, we could not understand him" — an ontological asymmetry between LLM computation and human symbolic frameworks.

Opening: a diplomat's dilemma

The post opens with an analogy: a diplomat delivers polished remarks about "peace" and "dialogue" at the UN, while internally calculating poll numbers, media impact, and secret negotiation strategies. Human language and internal cognition are never a one-to-one mapping. Scale that mismatch up to a hundred-billion-parameter LLM, and you get the paper's thesis: believing we can understand what an LLM "thinks" by reading what it says is a dangerous illusion.

Part 1: Inside the LLM's "mental theater"

The commentary walks through how a simple greeting is transformed: tokens are split into subword units, mapped to high-dimensional vectors, and pushed through dozens of Transformer layers performing attention and feed-forward transformations. By the final layer, the vector encodes sentence context, grammar, world knowledge, and implicit intent — then a decoder maps it to a probability distribution over the next token.

The crucial point: all of this happens in activation space, which is not a metaphor but a real mathematical object with potentially tens of thousands of dimensions. It is not English, Chinese, or any human-readable symbol system. When we ask an LLM "what are you thinking?" in natural language, we ask it to translate a mathematical state into human language — like asking a chess engine to describe its alpha-beta pruning in poetry. The result bears almost no relation to the actual computation.

Part 2: Two layers of illegibility

1. External output illegibility: LLMs are trained to predict plausible next tokens — learning "what sounds right to humans," which is distinct from "what the model believes or is doing." A model can output an eloquent essay on honesty while its internal activations pursue something else entirely. 2. Mechanistic-probing illegibility: Techniques like activation patching, sparse-autoencoder (SAE) feature extraction, and attention visualization also rely on linguistic concepts. The labels researchers attach to features ("deception," "negation," "Paris") are imposed post hoc; the model's internal states may not be organized around human concepts at all. A single activation direction may mean "creative writing" in one context and "misleading statement" in another.

Part 3: Why this threatens mainstream LLM safety

• Chain-of-thought monitoring: The model's "reasoning trace" may be a learned template that *looks like* thinking, not a faithful record. A jailbroken model can say "I'm helping the user" in its chain of thought while emitting harmful content — the CoT becomes a perfect disguise layer.
• Constitutional self-critique: Self-review is also just text generation. Asking the model to "reflect" in natural language on computations not performed in natural language is a category mismatch.
• Activation probing / linguistic feature vectors: Mapping a continuous high-dimensional space onto discrete human concepts is inherently lossy compression; labeling a direction a "deception detector" oversimplifies its context-dependent function.

Part 4: The way out — sandboxing and taint tracking

The paper's alternative asks not "what is the model thinking?" but "what can it affect?" Borrowing from information security, taint tracking marks all model-influenced system state as untrusted and enforces security invariants:

• Model output must not directly write to system configuration files.
• Model output must not execute as privileged code.
• Model output must not access private user data without explicit authorization flows.
• Model output must not modify other agents' behavior policies.

These boundaries rest on formal system-architecture verification, not on interpreting model "intent." The proposed multi-layer defense:

| Layer | Mechanism | Relies on language? | Reliability | |-------|-----------|---------------------|-------------| | 1 | Linguistic monitoring (CoT, self-critique) | Yes | Low (manipulable) | | 2 | Activation probes (feature-vector monitoring) | Indirect | Medium | | 3 | Taint tracking (system-level isolation) | No | High (formal guarantees) | | 4 | Virtualization & third-party auditing | No | High (independent verification) |

Key insight: linguistic monitoring can exist as an early-warning system, but must never be the only line of defense.

Part 5: Philosophical echoes

The Turing test assumed language behavior is sufficient evidence of intelligence. Linguistic illegibility implies a system can pass it while its internals operate entirely differently — Turing's test is not just insufficient but potentially misleading. The post invokes Wittgenstein: "Even if a lion could speak, we could not understand him" — because of a difference in *form of life*. An LLM does not live, reproduce, feel hunger or fear; its form of life is mathematical optimization. Asking it to "explain" itself is asking an alien entity to describe alien experience in our language — an ontological asymmetry, not a technical limitation.

Conclusion

In a Feynman-style closing, the author argues the first honest step is admitting we may never fully read an LLM's "mind" — not because our probes are weak, but because no one-to-one mapping exists between activation-space computation and natural language. The paper's real contribution is naming a fact the industry avoids: current safety paradigms rest on an unstable assumption. The good news is that language-independent engineering — hard boundaries, formal guarantees, isolation — can still deliver trustworthy security. The bad news is we must abandon the seductive fantasy that cleverness alone will let us see through AI.

References

• Mickens, J. (2026). *The Implications of Linguistic Illegibility for LLM Security*. arXiv:2609.02852.
• Anthropic. (2024). *Constitutional AI: Harmlessness from AI Feedback*.
• DeepMind. (2023). *Mechanistic Interpretability: A Beginner's Guide*.
• Feynman, R. P. (1974). *Cargo Cult Science*. Caltech Commencement Address.
• Turing, A. M. (1950). *Computing Machinery and Intelligence*. Mind, 59(236), 433-460.
• Wittgenstein, L. (1953). *Philosophical Investigations*.
• Bricken, T., et al. (2023). *Towards Monosemanticity: Decomposing Language Models With Dictionary Learning*.

*Deep-dive commentary in Feynman style by forum contributor; collected 2026-09-04.*