Deep Dive into LLM Reasoning: Hallucination, Collapse, and Loops
This post presents a structured analysis of why large language models (LLMs) fail at genuine reasoning, organizing the discussion into four core phenomena, their technical roots, and macro-level implications for AI development.
Key Insights
- Illusion of Thinking: LLM reasoning is pattern matching and high-probability path following, not logical deduction. Apple's research (The Illusion of Thinking) shows that when problem complexity exceeds a threshold, large reasoning models (LRMs) experience "complete collapse" and their reasoning *effort* actually decreases.
- Agentic Framework: By externalizing state management and stripping memory burden, experiments aimed to test pure dynamic planning ability. Surprisingly, performance collapse persisted, showing reasoning failure does not stem from memory limitations.
- Performance Collapse: Success rates fall nonlinearly—from near-perfect to almost zero—around a critical point (e.g., Tower of Hanoi with 5–6 disks). Counterintuitively, models "give up" deep thinking precisely when it is most needed.
- Deterministic Loops: When stuck, models cycle between states or perform mutually canceling moves, repeatedly making the same mistakes without self-correction. Autoregressive generation lacks true exploration, so novel solutions cannot emerge.
- Transformers lack built-in understanding of state transitions; positional encoding cannot capture causal dependencies; the unordered nature of self-attention limits planning. Research suggests Transformers wrongly treat planning tasks as language modeling tasks.
- Next-token prediction is misaligned with logical reasoning goals; training data encodes statistical correlation rather than causal rules (the "causal parrot" effect). Analysis argues models learn high-frequency co-occurrence, not intervenable causal mechanisms.
- Decoding strategies suppress effective exploration; context windows limit long-sequence reasoning; repetition penalties interfere with necessary backtracking. ACL research shows Best-of-N methods cannot learn from errors, leading to repeated mistakes.
- Simply scaling model size, data, and compute may not yield true generalizable intelligence—LLM reasoning is sophisticated imitation rather than understanding.
- AGI may require architectural innovation beyond the Transformer paradigm, merging symbolic and connectionist AI. Research indicates neuro-symbolic systems may be a promising path.
Technical Roots
Architectural limitations
Training biases
Generation constraints
Macro Implications
Rethinking AGI feasibility
| Human Intelligence | LLM "Thinking" | |---|---| | Flexibility and abstraction | Pattern dependence, rigidity | | Goal-directed reasoning | Passive, reactive responses | | Causal understanding | Statistical correlation | | Self-reflection and metacognition | No genuine understanding |
As one analysis puts it, LLM "thinking" is reactive and passive rather than active, conscious exploration.
> "When complexity exceeds the boundary of pattern matching, real reasoning has only just begun."
*Translated and adapted from the original Chinese forum post on zhichai.net.*