KisMATH Deep Dive: Do LLMs Actually Reason or Just Memorize?

> Paper: *KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?* > Authors: Soumadeep Saha, Akshay Chaturvedi, Saptarshi Saha, Utpal Garain, Nicholas Asher > Institutions: ISI Kolkata, IRIT, LINAGORA Labs > Published: Accepted to TACL, 2026 > arXiv: 2507.11408 > Datasets & code: https://kismath.github.io

1. The Core Question: Real Thinking or Pretend Thinking?

When DeepSeek-R1 or OpenAI o3 outputs long chains of "Let me think step by step...", is it performing genuine logical deduction, or is it a statistical machine imitating the posture of thought?

This debate has split the scientific community into two camps:

• Reasoning Hypothesis: CoT decomposes complex problems into subtasks, solves them step by step, and combines results. (Representatives: OpenAI o1/o3, DeepSeek-R1)
• Approximate Retrieval Hypothesis: CoT is just approximate retrieval from latent memory; reasoning traces are decorative. Evidence includes insensitivity to random digit replacement and perturbed in-context examples. (Representative: Kambhampati, 2024)

KisMATH offers new empirical evidence on this debate.

2. Methodological Innovation: Causal CoT Graphs (CCGraphs)

Previous CoT tests relied on random perturbation (replacing digits, shuffling examples), which cannot distinguish "model is insensitive to values" from "model doesn't depend on reasoning structure at all." The KisMATH team proposes: interventions aligned with the implicit structure of reasoning reveal more than random perturbation.

A Causal CoT Graph is a DAG automatically extracted from LLM reasoning traces:

• Nodes: mathematical expressions
• Edges: fine-grained causal dependencies

The extraction algorithm backtracks from the answer node, matching nodes via exact string match or shared parse-tree nodes, recursively extending until question nodes are reached, then inverting edges to form a DAG and pruning orphans. Only ~10% of nodes/edges required manual correction (88/40K nodes, 71/300K edges).

From each CCGraph, the longest Q→A path (R path) is extracted: top-5 for GSM8K, top-10 for MATH500/AIME, with 9-40 reasoning nodes and 6-10 reasoning hops per question.

3. Dataset and Models

| Dataset | Problems | Type | Difficulty | |---|---|---|---| | GSM8K | 983 | Grade-school arithmetic word problems | ⭐⭐ | | MATH500 | 384 | Olympiad-style math | ⭐⭐⭐⭐ | | AIME | 304 | Competition-level math | ⭐⭐⭐⭐⭐ | | Total | 1,671 | | |

CoT traces were generated with OpenAI o3-2025-04-16 (5-shot CoT), keeping only correct-answer traces. Testing covered 15 open-source LLMs (1B-70B): Gemma 3, Qwen 3, DeepSeek-R1, Llama 3.1/3.3, Qwen 2.5, and DeepSeek-R1 Distill. Compute: 4×A100, ~3,000 GPU-hours.

4. Experiments and Key Findings

4.1 Are reasoning nodes causal mediators?

Using attention suppression (zeroing attention to all reasoning-node tokens), answer entropy H(Pₐ) increased significantly (p < 10⁻¹²) with high Kolmogorov distances.

Conclusion: Reasoning nodes are causal mediators of the answer — a necessary condition for reasoning. This directly refutes the extreme "CoT is just decoration" view.

4.2 Counterfactual validation of R paths

Suppressing only R-path nodes also significantly increased answer entropy (KS test, very low p-values). Not all reasoning tokens matter equally — nodes on structured paths have special causal status.

4.3 Do LLMs internally realize CCGraph-like structure?

Comparing R-path probability against random paths of equal length (avoiding CCGraph nodes):

• A significant peak at the 100th percentile of rank distribution
• Consistent across all 15 LLMs and 3 splits (only Llama 3.3 70B on AIME was an exception)

Conclusion: LLMs assign significantly higher probability to causal reasoning paths — they internally encode CCGraph-like structure.

5. Reconciling Contradictory Results

| Approach | Method | Result | |---|---|---| | Random perturbation | 50% digit replacement | Performance unchanged — models are robust to values | | Structure-aligned intervention | Attention suppression on CCGraph nodes | Uncertainty spikes — models are sensitive to structure |

Unified hypothesis: LLMs likely exploit both structure and memory. Random digit replacement preserves the structural skeleton (which operations, in what order), so models can still follow the map; but destroying the structure itself leaves the model lost. This resembles human mathematicians: specific numbers can be swapped once the proof structure is understood, but deleting proof steps makes completion impossible.

6. Why CCGraph Beats Prior Methods

| Work | Scale | Approach | |---|---|---| | Tan (2023) | 27 GSM8K problems | Manual causal graphs | | Lee et al. (2025) | 30 | Rich edge types, computationally prohibitive | | Bogdan et al. (2025) | 10 | Rollout sampling, unreliable attention aggregation | | KisMATH | 1,671 | Automatic CCGraph + attention suppression — scalable, causally precise, structure-aware |

7. Limitations and Future Directions

Limitations: math-domain only (no code/logic/science), ~10% manual fixes for LaTeX errors, simplified DAG (no planning/backtracking/reflection edges), limited coverage of closed-source models, and static graphs extracted from complete traces.

Future directions: extending to programming (Codeforces), logic (LSAT), and science; dynamic CCGraphs updated during token generation; richer edge types; and applications in error detection, RL reward design, and structured reasoning visualization.

8. Core Conclusions

1. LLMs do perform causal reasoning: suppressing reasoning nodes significantly increases answer uncertainty — reasoning nodes are causal mediators. 2. LLMs internally implement implicit structure: R paths receive significantly higher probability than random paths. 3. Structured interventions reveal what random perturbation cannot: structure is key; specific values are secondary.

The study doesn't fully refute the approximate-retrieval view — LLMs may use both structure and memory. But it establishes that structured reasoning is a genuine mechanism behind CoT's performance gains, not pure decoration. When DeepSeek-R1 "thinks" on screen, it is performing some form of structured causal inference — not human-level understanding, but not pure recitation either. It is a cognitive phenomenon somewhere in between, and we are just beginning to understand its nature.

References

1. Saha S, Chaturvedi A, Saha S, Garain U, Asher N. *KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?* arXiv:2507.11408v2 [cs.CL], 2026. Accepted to TACL. 2. Wei J, et al. Chain-of-thought prompting elicits reasoning in large language models. *NeurIPS*. 2022;35:24824-24837. 3. Guo D, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948, 2025. 4. Kambhampati S. LLMs can't plan, but can help planning. *SSRN*. 2024. 5. Lanham T, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv:2307.13748, 2023.