Scaffolds and Skyscrapers: Mathematical Reasoning Isn't Trained by Writing Code
> *A commentary on 'What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code'*
The Architect and the Bricklayer
Imagine two architecture students. One spends three years on a construction site and becomes a master bricklayer—give him bricks and he'll build a perfect wall. The other spends a year studying structural mechanics, materials science, and architectural history; she has built few walls, but she understands *why* walls are built the way they are. Who designs a better skyscraper? The answer is obvious.
This paper is about the 'bricklayer vs. architect' problem in AI training: we assumed that having AI write more code makes it reason better, but the truth is that reasoning ability comes from training on structured thinking, not code itself.
The Problem
Paper: Yuze Zhao, Xinran Zhang, Guoqiang Xu, Haonan Wang, Yikang Shen, Yining Ye, Zhezheng Ying, Jiaya Jia, Jian Zhao, Jingdong Wang. *What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code*. arXiv:2605.19762, 2026.
Field: AI / large language models / mathematical reasoning / data engineering
In the AI community, there is a near-consensus belief: adding code to pretraining data makes LLMs better at reasoning. The rationale sounds plausible—code is a rigorously structured language; writing code requires step-by-step derivation, conditionals, and iteration; strong reasoning models like GPT-4 and Claude were indeed trained heavily on code. So companies have crawled GitHub and Stack Overflow and poured code into models as 'fertilizer for reasoning.'
But the paper asks a crucial, overlooked question:
> 'Is it code itself that improves reasoning, or something else inside code data?'
Background: The 'Black Box' of Pretraining Data
1. Pretraining = letting the model read a massive slice of the internet, learning from statistical patterns without an explicit curriculum. 2. 'Code data' is not a single category. It includes pure executable programs, code comments, Code-NL mixes (Stack Overflow Q&A), Math-Code mixes (algorithms in math papers), and Math-Text mixes (textbooks, derivations). 3. Confounding variables. Code data typically comes bundled with technical documentation, structured reasoning traces ('first... then... therefore...'), and mathematical content. These companions—not code itself—might be the true source of reasoning gains.
Experimental Design: Precise Variable Control
Instead of naively comparing 'with code vs. without code,' the researchers ran a large-scale 10T-token pretraining experiment with a corpus of fine-grained domain separation:
| Data type | Description | |-----------|-------------| | Pure code | Standalone executable programs | | Code-NL | Technical docs with explanations | | Math-Text | Derivations, theorem proofs | | Math-Text Mix | Textbooks, step-by-step solutions | | Math-Code Mix | Algorithms within math papers | | General | Wikipedia, news, etc. |
They systematically varied domain proportions and observed model performance across tasks.
Three Groundbreaking Findings
1. Pure code ≠ a universal reasoning enhancer
> When code is restricted to pure executable programs and Code-NL data is strictly controlled—code substantially improves programming ability but does not act as a general reasoning enhancer.
Reading more Python makes a model better at Python, but not better at math or logic. Worse, code training even competes with knowledge-intensive tasks, especially complex mathematical reasoning.
2. The real hero: structured reasoning traces
What makes 'code-trained' models *look* better at reasoning? Cross-domain structured reasoning traces—code-text and math-text mixed data—rather than executable code itself. These mixtures share explicit step-by-step reasoning scaffolds:
- 'First, we observe...'
- 'Next, assume...'
- 'It follows that...'
- 'Therefore, we conclude...'
3. Structured density within the math domain is key
> Within a fixed math budget, increasing the density of structured math samples yields significant gains on difficult math reasoning tasks while largely preserving coding performance.
Feeding the model 100B tokens of high-quality step-by-step math derivations beats 1000B tokens of miscellaneous code for math reasoning—without damaging other capabilities.
Mechanistic Evidence: Expert Activation Patterns
Routing analyses show that changes in data composition are reflected in the model's expert activation patterns: different 'experts' (subnetworks) activate when the model switches between code and math. This provides mechanistic evidence of competitive and synergistic interactions across domains—the model is not a uniform sponge; code training and math training compete for specialized regions.
Implications: Data-Centric Optimization
1. From volume to structure: carefully tuning the proportion of structured reasoning signals beats mindlessly stacking code data. 2. The 'scaffold' theory: learning complex skills requires practicing intermediate structures (the reasoning process), not just the final product (code)—like practicing scales before improvising, or dissecting logic before writing. 3. The Feynman lens: like Feynman's Brazilian students who could recite physics terms without understanding phenomena, a model can 'recite' code (generate syntactically correct programs) without truly learning to think if its data lacks explicit reasoning structure—a superb bricklayer, not an architect.
Why This Paper Matters
1. It punctures a myth: rigorous experiments overturn the 'code = reasoning' industry belief. 2. It points the way: the key to reasoning gains is structured reasoning data, not code volume. 3. Practical value: it offers concrete data-mixing strategies directly relevant to training the next generation of models.
Reference
Zhao, Y., Zhang, X., Xu, G., Wang, H., Shen, Y., Ye, Y., Ying, Z., Jia, J., Zhao, J., & Wang, J. (2026). *What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code*. arXiv preprint arXiv:2605.19762.
---
*'Lay ten thousand walls and you remain a bricklayer; understand the mechanics of one wall and you become an architect.'*