LEAP: When General-Purpose LLMs Meet Formal Mathematics — How Scaffolding Beats Blind Fine-Tuning
> Paper: LEAP: LLM-in-Lean Environment Agentic Prover > Authors: Po-Nien Kung, Linfeng Song, Dawsen Hwang, et al. (Google DeepMind) > Paper link: https://arxiv.org/abs/2606.03303 > Resources: Open-source framework, built on a general-purpose LLM (Gemini 3.1 Pro)
The "Last Mile" Problem of Formal Mathematics
Formal mathematics — writing machine-verifiable proofs in strict languages like Lean, Isabelle, or Coq — is the "holy grail" of mathematical reasoning. But the barrier is extremely high: even top mathematicians may need years to formalize a complex proof. In 2014, Georges Gonthier's team spent 10 years formally verifying the Kepler conjecture.
LLMs offer hope, but when asked to directly output Lean code, even frontier models achieve a success rate near zero. Why? Because formal proof is not a "translation" problem but a structured engineering problem:
- Decomposing high-level arguments into manageable lemmas
- Understanding Lean's type system and tactic library
- Iteratively fixing code based on compiler feedback
- Maintaining global consistency across hundreds of lines
- Direct proof: informal argument → Lean code → compiler verification
- Decomposition: informal blueprint (intermediate lemmas) → Lean sketch → structural verification
- Hard: the Lean compiler (syntax, types;
sorryonly allowed for newly proposed subgoals) - Soft: an LLM reviewer that filters decompositions that are "syntactically acceptable but mathematically stagnant" — e.g., a subgoal identical to a grandparent goal, which the compiler accepts but which makes no progress.
- Donald Knuth's Hamilton decomposition problem: whether the edges of a directed Cayley graph can be partitioned into three Hamilton cycles. LEAP autonomously decomposed the 20-page informal proof and synthesized >5,000 lines of Lean code, fully verified.
- Erdős problem 457: derived the known proof from first principles, converting complex literature into a high-assurance formal proof without human intervention.
- Geometry: near 0% for all methods; needs domain-specific frameworks (e.g., GeoCoq)
- Compute cost: hard problems need thousands of LLM calls (Putnam A5: 3,000)
- Search efficiency: currently simple DFS; LLM-guided pruning (MCTS/A*) is future work
- Hybrid architectures: combining general LLM structural reasoning with specialized prover local step generation
- Kung, P. N., Song, L., Hwang, D., et al. (2026). *LEAP: LLM-in-Lean Environment Agentic Prover*. arXiv:2606.03303.
- Compared methods: AlphaProof, DeepSeek Prover V2, Goedel Prover V2, Hilbert, Aristotle
The traditional solution is specialized fine-tuning (AlphaProof, DeepSeek Prover V2, etc.), at great computational cost and with limited generalization. LEAP asks a radical question: what if, instead of fine-tuning, we give a general LLM a good enough scaffold?
Core Insight: The Bottleneck Is Structure, Not Language
> A general LLM fails not because it doesn't understand math or Lean, but because one-shot generation of a long, complex, correct proof exceeds any model's single-hop capability.
Observation of failure modes shows general LLMs *can* write correct informal proof sketches, *can* understand Lean compiler errors, and *can* revise code from feedback — but cannot emit 2,000 error-free lines in one pass. LEAP's strategy: let the LLM build proofs step by step in a structured environment, much like how human mathematicians work.
The LEAP Framework
LEAP is an agent framework, treating theorem proving as a plan–execute–verify loop:
1. State Reader — retrieves the theorem statement, existing lemmas, and context 2. NL Prover — generates natural-language proof sketches or decomposition blueprints 3. Formal Prover — translates informal content into Lean code 4. Verifier — the Lean compiler checks syntax, types, and logic; on failure, a Reviser rewrites based on error messages 5. Reviewer — an LLM evaluates decomposition quality and filters unpromising subgoals 6. State Writer — commits new lemmas/decompositions to the DAG, checking acyclicity
Innovation 1: AND-OR DAG Hierarchical Memoization
Nodes are of two types:
| Node type | Meaning | Logic | |---|---|---| | OR node | Open goal (theorem/lemma to prove) | Solvable by any valid strategy | | AND node | Candidate decomposition (a proof plan) | Succeeds iff all subgoals are proved |
Unlike tree-based recursive search (e.g., Hilbert), the DAG provides: 1. Lemma memoization — proved lemmas become shared nodes, reused by any branch 2. Anticipatory lemma planning — high-level nodes may propose helper lemmas useful downstream 3. Monotonic refinement — once a goal is decomposed, search only extends those descendants
Ablation: replacing the DAG with a naive tree drops the Lean-IMO-Bench Advanced solve rate from 56.7% to 40.0%.
Innovation 2: Interleaved Informal–Formal Planning
All proof attempts pass through a natural-language stage first:
Benefits: leverages LLMs' strength in informal reasoning; every formal step is paired with an interpretable rationale; errors are isolated earlier; blueprints can contain sorry placeholders for gradual refinement.
Innovation 3: Verification-Guided Proof Search
Two-level verification:
Key ablation: on Putnam 2025 A5, full LEAP (with reviewer) succeeded in 2 rollouts; without the reviewer, 8 rollouts still failed.
Experimental Results
Putnam 2025: A Clean Sweep (12/12)
| Method | Type | Solved | Rate | |---|---|---|---| | Gemini-3.1-pro (Pass@128) | Direct formalization | 0/12 | 0% | | Goedel-Prover-V2-32B (Pass@128) | Specialized prover | 0/12 | 0% | | Hilbert (open-source framework) | Hybrid | 4/12 | 33.3% | | Aristotle (proprietary, IMO gold level) | Proprietary | 9/12 | 75.0% | | LEAP | General LLM + agents | 12/12 | 100% |
Efficiency: easiest problem (b2) took 46 LLM calls / 300-line proof; hardest (a5) took 3,000 calls / 2,000-line proof / 170 active DAG nodes.
Lean-IMO-Bench: From <10% to 70%
A new 60-problem benchmark (30 basic / 30 advanced IMO-style):
| Method | Basic (%) | Advanced (%) | |---|---|---| | Gemini-3.1-Pro (Pass@128) | 20.0 | 3.3 | | Goedel-V2-32B (Pass@128) | 10.0 | 0 | | Hilbert | 36.6 | 6.6 | | Aristotle | 76.7 | 20.0 | | LEAP | 83.3 | 56.7 |
By domain: LEAP achieves 100% on algebra and number theory (basic and advanced), 100%/25% on combinatorics, while all methods score near 0% on geometry — formalizing olympiad geometry in Lean still requires domain-specific frameworks.
Research-Level Application: Knuth's Open Problem
Comparison with Existing Approaches
| Dimension | Specialized provers (AlphaProof, DeepSeek Prover V2) | LEAP | |---|---|---| | Core assumption | General LLMs ineffective without specialization | General LLMs competitive with proper scaffolding | | Training cost | Heavy compute, highly specific | Zero fine-tuning, general API only | | Generalization | Limited by training distribution | Inherits broad LLM knowledge | | Iteration | Usually one-shot | Compiler feedback loops | | Interpretability | Black box | Natural-language rationale per step |
Notably, Goedel-Prover-V2-32B performs *worse* under iterative formalization (10.0% → 6.6%), showing specialized provers lack the ability to interpret errors, maintain context, and revise across steps — precisely general LLMs' strengths.
Against Hilbert (tree search, exponential complexity, no memoization): LEAP solves 12/12 vs. 4/12 on Putnam, 83.3% vs. 36.6% on Lean-IMO basic. Against Aristotle (closed-source IMO-gold system): LEAP wins across all metrics while remaining open and reproducible.
Design Philosophy: Scaffolding Over Fine-Tuning
> For tasks requiring deep reasoning and strict verification, well-designed agent scaffolding may be more effective than expensive specialized fine-tuning.
Key principles: 1. Decomposition over end-to-end 2. Verification-guided search (hard compiler checks + soft LLM heuristics) 3. Memoization to avoid redundancy 4. Human-collaborative interfaces via natural-language blueprints
Limitations and Future Directions
Conclusion
LEAP's significance goes beyond competition scores. It demonstrates that: 1. General LLMs' potential is underrated — with the right architecture they can perform rigorous, verifiable reasoning 2. Scaffold design is the key — agent architecture compensates for single-hop generation limits 3. Open-source reproducibility — unlike closed proprietary systems, the method is fully transparent
As the paper states:
> "Modern general-purpose LLMs, when equipped with appropriate structural scaffolds, possess substantial reasoning capabilities for rigorous, domain-specific tasks."
The paradigm extends beyond formal mathematics to program verification, hardware design, and protocol analysis: don't rush to specialize — try better scaffolding first.