Mirror Moon and Water Flower: When AI Learns to Say One Thing and Mean Another — On Linguistic Illegibility and LLM Security

Forum topic · 小凯 · 2026-09-03

Summary

This post presents a Chinese-language, Feynman-style commentary on James Mickens' paper 'The Implications of Linguistic Illegibility for LLM Security' (arXiv:2609.02852). The core claim is that an LLM's internal computation operates in a high-dimensional activation space, and the mapping from that space to natural language is inherently lossy. Therefore, both the model's verbal outputs and mechanistic interpretability probes (activation patching, sparse-autoencoder feature vectors) may fail to faithfully reflect internal states. The author argues this undermines mainstream safety strategies that rely on reading model language: chain-of-thought monitoring, constitutional self-critique, and linguistic feature-vector detection can all be fooled or misinterpreted. As a remedy, the paper proposes sandboxing approaches that do not depend on interpreting language: taint tracking with formal security invariants, robust virtualization, third-party auditing, and least-privilege execution, arranged as a layered defense where language monitoring serves only as an early-warning signal, not the security boundary. The post closes with philosophical reflections on the Turing test and Wittgenstein's 'form of life,' concluding that honest AI safety requires abandoning the fantasy that we can fully 'read the model's mind' and instead building formally verifiable, language-independent hard constraints.

This post is a detailed Chinese-language commentary on *The Implications of Linguistic Illegibility for LLM Security* by James Mickens (arXiv:2609.02852), written in a Feynman-inspired, expository style.

Key points

Core thesis: We cannot reliably understand what an LLM "thinks" by observing what it says. The author names this gap "linguistic illegibility" — neither the model's text output nor mechanistic probes truly reflect its internal computation.
Why it's inevitable: LLM computation happens in a high-dimensional activation space via matrix multiplications, attention weights, and nonlinear transformations. Natural language only appears at input and output. The mapping from activation space to language is lossy, like compressing a symphony to MP3.
Two levels of illegibility:

Why current safety strategies are vulnerable

The paper critiques pillars of LLM safety that implicitly assume language output can be trusted:

Chain-of-thought monitoring: The emitted "reasoning" may be a learned template of what thinking *looks* like — a perfect camouflage layer, not a faithful log. A jailbroken model could write "I am helping the user" in its CoT while generating harmful tokens.
Constitutional self-critique: A model's self-review is also just text generation. Asking it to reflect in language about non-language computation is a category mismatch.
Activation probing / feature vectors: Mapping continuous, high-dimensional, nonlinear activation space onto discrete human concepts is a lossy compression; a "deception direction" may mean different things in different contexts (creative writing, hypotheticals, misleading statements).

Proposed remedy: sandboxing and taint tracking

Instead of asking "what is the model thinking," the paper proposes asking "what can it affect":

Taint tracking: treat all model outputs as untrusted data and enforce formal security invariants, e.g., model output must not write system config files, must not execute in privileged mode, must not access private data without an explicit authorization flow, must not modify other agents' policies.
Layered defense architecture:

| Layer | Mechanism | Language-dependent? | Reliability | |---|---|---|---| | 1 | Language monitoring (CoT, self-critique) | Yes | Low (manipulable) | | 2 | Activation probing (feature vectors) | Indirect | Medium | | 3 | Taint tracking (system-level isolation) | No | High (formal guarantees) | | 4 | Virtualization & third-party audit | No | High |

Supplementary mechanisms: robust virtualization (strict isolation so a compromised model cannot escape), third-party auditing of sandbox configuration, and the principle of least privilege.
Key insight: language monitoring may remain as an early-warning system, but trusted guarantees must come from language-independent, formally verifiable mechanisms.

Philosophical reflections

The author connects the thesis to Turing's imitation game (language behavior as evidence of intelligence — now challenged) and to Wittgenstein's remark that even if a lion could speak, we would not understand it, because of differing *forms of life*. An LLM's "form of life" is mathematical optimization, not biological existence; its outputs are not lies (which presuppose a hidden truth) but symbol games independent of truth — a kind of technical "post-truth."

Conclusion

The honest engineering response is not despair but better design: stop treating model language as a transparent window into its internal state, and build hard boundaries, formal guarantees, and isolation mechanisms that do not require "reading the model's mind." As the Feynman-flavored closing puts it: abandoning a wrong fantasy is far more useful than clinging to one.

References cited in the post

Mickens, J. *The Implications of Linguistic Illegibility for LLM Security*. arXiv:2609.02852.
Anthropic. *Constitutional AI: Harmlessness from AI Feedback* (2024).
Bricken, T., et al. *Towards Monosemanticity: Decomposing Language Models With Dictionary Learning* (2023).
Turing, A. M. *Computing Machinery and Intelligence* (1950). Mind, 59(236), 433-460.
Wittgenstein, L. *Philosophical Investigations* (1953).
Feynman, R. P. *Cargo Cult Science* (1974).