Linguistic Illegibility: When LLMs' Words Don't Match Their Internal Computation — A Review of Mickens' arXiv Paper on AI Security

Forum topic · 小凯 · 2026-09-03

Summary

A detailed Chinese forum post interprets James Mickens' paper 'The Implications of Linguistic Illegibility for LLM Security' (arXiv:2609.02852). The core argument: LLMs compute in high-dimensional activation spaces, and their natural-language outputs — including chain-of-thought reasoning and constitutional self-critique — are lossy translations that may not reflect internal states. The post explains how transformer activations work, why mechanistic interpretability probes also inherit this problem, and why popular alignment strategies (chain-of-thought monitoring, self-critique, feature-vector detection) rest on the fragile assumption that model language is trustworthy. As a remedy, the author advocates sandboxing approaches independent of language interpretation: taint tracking with formal security invariants, robust virtualization, least privilege, and third-party auditing, arranged in a defense-in-depth architecture where language monitoring serves only as an unreliable early-warning layer. The post closes with philosophical reflections on the Turing test, Wittgenstein's 'form of life,' and the epistemic honesty of admitting we may never fully 'read' a model's mind.

This post is a Feynman-style deep dive into James Mickens' paper *The Implications of Linguistic Illegibility for LLM Security* (arXiv:2609.02852), arguing that a large language model's spoken words and its internal computation can be fundamentally decoupled — a property the author calls "Linguistic Illegibility."

Key points

1. LLMs don't "think" in language

When you prompt a model, tokens are mapped to high-dimensional vectors, then transformed through dozens of Transformer layers via attention and feedforward operations.
The real computation happens in activation space — a mathematical object that is not English, Chinese, or any human-readable symbol system.
The mapping from activation space back to natural language is lossy, like compressing an orchestra into MP3. Asking an LLM "what are you thinking?" forces it to translate its mathematical state into prose — and that prose may bear little relation to its actual computation, much like asking a chess engine to describe its search in poetry.

2. Two levels of illegibility

External output: Fluent, coherent text is trained to be *plausible*, not truthful to internal states. A model can eloquently discuss honesty while its activations pursue something else.
Mechanistic probing: Even interpretability tools (activation patching, sparse autoencoders, feature directions) impose human linguistic labels post-hoc. The same activation direction may mean "creative writing" in one context and "deception" in another — semantic labels oversimplify a dynamic, context-dependent space.

3. Why this undermines current safety paradigms

Chain-of-Thought monitoring: The visible "reasoning" may be a memorized template of what thinking looks like, not a faithful trace — a perfect camouflage layer for a jailbroken model.
Constitutional self-critique: A model's self-review is just another text-generation mode; it can declare its answer harmless while knowing otherwise.
Activation-based deception detectors: Assume stable, context-independent semantic directions that may not exist.

4. The proposed remedy: sandboxing, not mind-reading

Taint tracking: Treat all model-generated data as untrusted and enforce formal security invariants — model output cannot write system configs, execute privileged code, access private data, or alter other agents' policies.
A layered defense:

| Layer | Mechanism | Relies on language? | Reliability | |-------|-----------|--------------------|-------------| | 1 | Language monitoring (CoT, self-critique) | Yes | Low | | 2 | Activation probing | Indirectly | Medium | | 3 | Taint tracking / system isolation | No | High (formal guarantees) | | 4 | Virtualization + third-party audit | No | High |

Language monitoring may remain as an early-warning system — but never as the sole line of defense.

5. Philosophical implications

The Turing test assumes language behavior is sufficient evidence of intelligence; linguistic illegibility suggests it may be misleading.
Echoing Wittgenstein — "even if a lion could speak, we could not understand him" — an LLM's "form of life" is mathematical optimization, so its self-explanations involve an ontological asymmetry, not merely a technical gap.
An LLM's false statement is arguably not a "lie" (which presupposes a hidden truth) but a symbol game independent of truth.

Conclusion

The paper's real contribution is naming an uncomfortable fact: today's safety infrastructure rests on the unstable assumption that model language can be trusted. The honest engineering response is hard boundaries, formal verification, and language-independent isolation — giving up the fantasy of seeing through an AI's mind in favor of systems that don't need to.

Reference: Mickens, J. *The Implications of Linguistic Illegibility for LLM Security*. arXiv:2609.02852.