When Code Starts to Dream: The Hidden World of LLM Reasoning

This post presents an in-depth Chinese-language walkthrough of the 2025 research paper “Cognitive Foundations for Reasoning and Their Manifestation in LLMs” (arXiv:2511.16660), a collaboration between the University of Illinois, University of Washington, Princeton, and Harvard. Below is a structured English translation of the key content.

The Core Paradox

LLMs can pass the bar exam and solve complex science problems, yet fail on simple variants of those problems — suggesting they rely on sophisticated pattern matching rather than human-like understanding. To investigate, the researchers acted as “digital archaeologists,” mining 171,485 reasoning traces from modern AI systems as “thought fossils,” and constructed a bridge between cognitive science and AI research: a fine-grained taxonomy of 28 cognitive elements.

Part 1: The 28 Cognitive Elements — The DNA of Thought

The taxonomy is organized along four dimensions:

Reasoning Invariants (the “constitution” of thought)

• Logical consistency: Models frequently attempt to maintain consistency but often fail — they can detect contradictions but cannot resolve them like humans, an “execution gap.”
• Compositionality: Models try to combine concepts but often produce shallow, seemingly-plausible combinations (“blue sadness,” “square circle”).
• Productivity: Despite near-infinite data, models exhibit “refined repetition” rather than creative generativity.
• Conceptual processing: Models manipulate words but struggle to reach the abstract concepts behind them — a cognitive glass ceiling.

Metacognitive Control (the “operating system” of thought)

• Self-awareness: Only 16% of LLM research papers address it; in actual behavior it is nearly absent. Models cannot judge whether they truly understand a problem — a major safety risk.
• Contextual awareness: Expected in 70% of papers, but models adapt stiffly across domains and contexts.
• Strategy selection: Mentioned in 60% of papers, yet models tend to lock into a single mode — “strategy rigidity.”
• Goal management: Models track nested goals poorly, drifting off-topic mid-reasoning.
• Evaluation: Seriously explored in only 8% of LLM research, despite strong correlation with reasoning success. Models generate coherent-looking chains but cannot judge whether those chains lead off a cliff.

Reasoning Representations

• Ordinal, causal, and network organization: Causal links in models tend to be surface-level statistical associations rather than mechanistic understanding; models navigate knowledge networks randomly rather than purposefully.

Reasoning Operations

• Selective attention: Mechanical, unable to dynamically adjust focus.
• Decomposition and integration: 60% of research emphasizes decomposition, but models show “integration deficiency” — they can split problems but cannot reassemble solutions.
• Backtracking: Phrases like “wait, let me rethink” are often surface gestures — “pseudo-backtracking” that never truly reverts erroneous paths.

Part 2: 171,485 Thought Fossils

The dataset includes:

• 14 open-source text models (8B to 671B parameters, e.g., Qwen3, Llama, DeepScaleR, DeepSeek-R1) on 10,612 problems from GeneralThought and ClaimSpect
• 18,000 visual reasoning traces (math, physics, chess from images)
• ~5,000 audio reasoning traces (long conversations)
• 54 human think-aloud protocols with conservative (“minimal pooling”) annotation standards

A two-layer attention scan first scores each passage for information density, \(ID(S) = \frac{MI(S,Q)}{length(S)}\) (mutual information with the query divided by length), then performs deep analysis of high-density passages.

Using an extended Jonassen problem taxonomy, accuracy degrades along the well-structured → ill-structured spectrum:

• Algorithmic problems: 63.8% accuracy
• Rule-using problems: 54.4%
• Decision-making, troubleshooting, diagnosis: 44–55%
• Dilemma problems: paradoxically up to 82.4% for some models, since no objective ground truth is required

Part 3: Humans vs. LLMs — Two Cognitive Architectures

• Hierarchy vs. flatness: 70% of human traces show clear parent–child goal structures; only 30% of LLM traces do, and mostly as superficial “step 1, step 2” sequences.
• Metacognitive monitoring vs. blind cruising: Self-evaluation occurs ~8.3 times per 1,000 words in human traces vs. 1.2 in LLM traces, mostly surface-level gestures. Humans backtrack and restructure globally; models patch locally.
• Frequency–success mismatch: On algorithmic problems, logical consistency (82% occurrence) and compositionality (76%) have low success correlation (0.08, 0.12), while knowledge alignment (34% occurrence) reaches 0.31. On dilemmas, sequential organization (91% occurrence) correlates only 0.15 with success, while hierarchical organization (22% occurrence) reaches 0.28 and representational restructuring (18%) reaches 0.33. Models spend effort on familiar, easily quantified behaviors rather than effective ones — partly because RLHF rewards plausible-looking outputs.

Part 4: The Research Community's Blind Spots

Analyzing 1,598 arXiv papers on LLM reasoning:

• Well-covered: sequential organization (55%), decomposition (60%), contextual awareness (70%)
• Neglected: self-awareness (16%), evaluation (8%), spatial organization (10%), temporal organization (22%)

This creates a self-reinforcing loop: easy-to-quantify behaviors get published → community follows → benchmarks fixate → models optimize for those benchmarks → an illusion of capability. The field may be optimizing the wrong targets — like searching for keys under the streetlight.

Part 5: The Miracle of Cognitive Guidance

The authors propose test-time reasoning guidance: automatically extracting consensus cognitive sub-graphs from successful human traces per problem type, linearizing them into structured “thinking scripts,” and converting them into natural-language prompts.

Results:

| Model | Dilemmas | Case analysis | Diagnosis | Avg. gain | | :--- | :--- | :--- | :--- | :--- | | Qwen3-14B | +60.0% | +44.0% | +56.0% | +32.0% | | R1-Distill-Qwen-32B | +60.0% | +48.0% | +36.0% | +40.0% | | Qwen3-32B | +48.0% | +41.9% | +24.0% | +32.0% | | R1-Distill-Llama-70B | +54.1% | +48.0% | +36.0% | +36.0% |

Key insight: LLMs possess deep reasoning capabilities that are “sealed” by default. Large models (32B+) benefit most; small models (<8B) sometimes degrade from information overload. The bottleneck is not parameter count but how latent capabilities are activated and organized — analogous to education unlocking latent potential.

Part 6: The Road Ahead

Three open challenges: 1. A predictive theory of training → capability: why RL cultivates verification but not metacognitive monitoring; possibly a dedicated “metacognitive RL” rewarding process reflection. 2. Cross-distribution generalization: structured training with diverse surface forms, explicit contrastive learning, and reflective prompting. 3. From behavior to mechanism: distinguishing genuine metacognition from imitation via systematic transfer, robustness to perturbation, compositional deployment, and internal consistency.

Conversely, LLMs become a large-scale experimental platform for cognitive science — testing working-memory limits, schema transfer, and metacognitive emergence in ways traditional psychology cannot.

Conclusion

LLMs are simultaneously imitators of reasoning and latent masters of it. The 28-element taxonomy is not a measuring stick for judgment but a searchlight for potential. The 60% gains from cognitive guidance are a starting point: the field should shift from a race of parameter scale to the refinement of cognitive architectures — from “big data + big compute” to “structure guidance + metacognitive activation.”

References

1. Kargupta, P., et al. (2025). *Cognitive Foundations for Reasoning and Their Manifestation in LLMs*. arXiv:2511.16660v1. 2. Jonassen, D. H. (2000). *Toward a design theory of problem solving*. Educational Technology Research and Development, 48(4), 63-85. 3. DeepSeek-AI, et al. (2025). *DeepSeek-R1: Incentivizing Reasoning Capability in Large Language Models via Reinforcement Learning*. 4. Fodor, J. A., & Pylyshyn, Z. W. (1988). *Connectionism and cognitive architecture: A critical analysis*. Cognition, 28(1-2), 3-71. 5. Lightman, H., et al. (2023). *Let's Verify Step by Step*. arXiv:2305.20050.

---

*This post is based on the paper “Cognitive Foundations for Reasoning and Their Manifestation in LLMs”; all data and viewpoints derive from that research and its cited literature.*