The Alchemy of Thinking: What Does AI Actually Learn When It 'Thinks' to Itself?
> "Thinking is the most powerful way to learn, but also the most dangerous—because you can never be sure whether what you've thought up is gold or iron pyrite."
This article analyzes the paper *Cognitive Mapping of Learning by Thinking Mechanisms in Large Language Models* (Han, Zhang & Yang, arXiv: 2611.14382), which proposes a Cognitive LbT framework mapping four human Learning-by-Thinking mechanisms onto LLM computational processes—and reveals four deep structural asymmetries that are not fixable bugs but fundamental architectural differences.
Key points
Mechanism 1: Reflective Reasoning (human self-explanation → LLM CoT / Self-Reflection)
- Chain-of-Thought (CoT) and Self-Reflection let models improve answers through intermediate reasoning and self-critique.
- Critical flaw: the presence of a chain of thought does not guarantee it is *faithful* to the model's actual computation—the stated reasoning may be post-hoc rationalization.
- The model is both generator and monitor: systematic errors cannot be caught by a self-check mechanism built from the same parameters.
- Self-Instruct (Wang et al., 2023) generated ~52,000 instructions from 175 seed tasks, achieving a 33% absolute improvement on Super-NaturalInstructions.
- Failure mode — Model Collapse: recursive training on self-generated data erodes distribution tails (Shumailov et al., 2024, *Nature*), like photocopying a photocopy.
- Hallucination diffusion: hallucinations become encoded as systematic features in parameters, not occasional errors.
- Self-Consistency samples multiple reasoning paths and votes; Tree-of-Thoughts searches branching reasoning paths.
- Fundamental difference: human simulation is *sensorimotor* and grounded in bodily experience; LLM simulation is purely *symbolic*, manipulating token probabilities without physical intuition—prone to plausible-but-wrong reasoning in long-tail physical scenarios.
- Risk of concept drift when internal simulation runs without external anchoring.
- Models spontaneously form semantic hierarchies during pretraining and self-supervised fine-tuning.
- Risk: self-organization without external teachers tends toward confirmation bias—bootstrapping new "knowledge" from old, with nothing to break the loop.
- P1: Self-reflection helps least precisely on tasks with systematic bias.
- P2: Recursive self-training shows an inverted-U performance curve; beyond a critical point, more self-training degrades performance.
- P3: Internal simulation benefits symbolic tasks (math) more than physically intuitive tasks (spatial reasoning).
- P4: Marginal gains from added reasoning steps diminish toward zero past a threshold while costs grow linearly.
- CoT may be rationalization rather than a faithful report: improving models based on such explanations risks "building on sand."
- Three of the four asymmetries relate to grounding—a problem of the *nature* of data, not its quantity, so scaling alone cannot fix it.
- Related approaches (evidence-chain memory, rubric-guided learning signals) all address the same question: how can AI reliably learn from itself? Closed-loop learning requires some form of external anchoring to prevent drift.
- Practical takeaways: don't treat all "self-learning" as the same; match LbT mechanisms to task types (CoT/ToT for symbolic reasoning, retrieval augmentation as artificial anchors, diversity constraints against confirmation bias); and stay skeptical of AI confidence, since self-checks share the blind spots of self-generation.
- Han, D., Zhang, J., & Yang, K. (2026). Cognitive Mapping of Learning by Thinking Mechanisms in Large Language Models. arXiv:2611.14382 [cs.AI].
- Wang, Y., et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL 2023.
- Shumailov, M., et al. (2024). AI models collapse when trained on recursively generated data. *Nature*.
- Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.
- Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. NeurIPS 2023.
- Zelikman, E., et al. (2022). STaR: Bootstrapping Reasoning with Reasoning. NeurIPS 2022.
- Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
- Shinn, N., et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023.
Mechanism 2: Synthetic Memory Formation (human constructive memory → LLM Self-Instruct / STaR)
Mechanism 3: Mental Simulation (human embodied rehearsal → LLM Self-Consistency / Tree-of-Thoughts)
Mechanism 4: Conceptual Self-Organization (human conceptual change → LLM embedding structure formation)
The Four Structural Asymmetries
1. Generator–monitor identity: LLMs use the same parameters to produce and to check reasoning, so systematic blind spots cannot be self-detected. 2. Closed-loop learning lacks external anchoring: human thinking is constantly interrupted by reality; LLM self-training loops can run perfectly sealed—perfectly heading toward collapse. 3. Symbolic vs. sensorimotor simulation: LLM "intuition" lacks grounding in physical experience; it "knows" glass shatters from text statistics, not perception. 4. Inference-time compute cost: LLM thinking cost grows nonlinearly with reasoning depth (CoT tokens, Self-Consistency samples, ToT branching), imposing hard feasibility constraints on reasoning depth.