Extracting Search Trees from LLM Reasoning Traces Reveals Myopic Planning

In May 2026, Chen et al. extracted and quantified search trees from LLM reasoning traces in the game of Connect Four, revealing a deep characteristic of LLM planning: although models generate extensive deep forward-looking analysis in their chain-of-thought (CoT), their actual move choices are best explained by a myopic model that ignores these deep nodes.

Key points

• Method: The pipeline parses LLM CoT text via natural language parsing, extracts a search tree (nodes = board states, edges = moves), fits computational models, and quantifies structural features such as search depth (max tree depth), search breadth (average branching factor), and node expansion patterns.
• Core findings:
• Move choices are best explained by a myopic model that ignores deep nodes.
• Win rate is predicted by search breadth, not depth.
• Deep nodes in the CoT appear decorative rather than decision-relevant.
• Causal intervention experiments:
• Pruning deep CoT segments (keeping only shallow nodes) leaves move quality nearly unchanged → deep nodes do not participate in decisions.
• Pruning shallow segments (keeping only deep nodes) significantly degrades move quality → shallow nodes are the decision core.
• Human vs. LLM contrast: Human experts rely on deep search as the core basis for decisions; LLMs rely on shallow breadth heuristics. Surface behavior looks similar, but underlying mechanisms differ.
• "Performative thinking" hypothesis: Deep CoT analysis in LLMs may be a statistical reproduction of "expert analysis patterns" from training corpora—pretraining supplies the textual form of deep analysis, SFT imitates the form without the function, and RLVR may reinforce shallow shortcuts.

Implications for test-time scaling

| Current assumption | Revised understanding | |:---|:---| | Longer CoT = deeper thinking | Length ≠ depth | | More test-time compute = better decisions | If deep nodes don't participate, extra compute may be wasted | | Explicit lookahead = real planning | Formal similarity ≠ functional equivalence |

The paper proposes new evaluation dimensions: actual decision depth (performance change after pruning different depths), structure–function consistency, and intervention robustness.

Relation to related work

• Coupling Tax: long reasoning chains carry hidden costs; this study shows much of a long chain may not participate in decisions at all.
• 80/20 Rule: decisions may depend only on the first ~20% of the CoT, with the rest being performative filler.
• Prefix Consistency: the intervention experiments act as a depth-selective perturbation.

Limitations and future directions

• Domain generalization: validated on Connect Four; extensions to chess/Go, mathematical proof, and program synthesis are needed.
• Model scale: does a larger model plan more deeply? Comparing specialized reasoning models vs. general models.
• Training interventions: deep-pruning penalties, mandatory citation of deep analysis, hierarchical attention mechanisms.
• Human–AI alignment: division of labor between myopic AI and deep-planning humans; risks of myopic planning in high-stakes settings.

Conclusion

1. Form–function dissociation: CoT contains the form of deep lookahead, but decisions are driven by shallow nodes. 2. Human–AI difference: human experts depend on deep search; LLMs depend on breadth heuristics. 3. Causal evidence: intervention experiments rule out the alternative explanation that deep analysis is useful. 4. Methodological contribution: the search-tree extraction framework generalizes to other strategic domains.

As test-time scaling is widely treated as the core strategy for improving reasoning, this work calls for more careful evaluation of whether "more thinking" truly translates into "better decisions."

Paper details

| Item | Content | |:---|:---| | Title | Extracting Search Trees from LLM Reasoning Traces Reveals Myopic Planning | | Authors | Sixing Chen, Ji-An Li, Saner Cakir, Sinan Akcali, Kayla Lee, Marcelo G. Mattar | | Affiliation | New York University, et al. | | arXiv ID | 2605.06840 | | Date | 2026-05-07 |