Overview
- Field: NLP
- Authors: Lingkai Kong, Zijian Wu, Yuzhe Gu, Haiteng Zhao, Wenyong Huang
- Published: 2026-07-13
- arXiv: 2607.11849
- ProverBench: the core proof-generation benchmark, containing 296 problems at undergraduate and doctoral qualifying-exam levels.
- Automatic verification pipeline: trained on large-scale expert annotations, it outputs correctness verdicts and fine-grained assessments of proof errors, and shows high agreement with human experts on held-out proof traces.
- VerifierBench: 888 model-generated proof traces paired with expert ground truth, testing whether models can correctly judge proof validity and provide sound verification rationales.
- AdvancedMathBench remains challenging for frontier models.
- Best proof-generation model GPT-5.5-xhigh scores only 75.8 (UGD split) and 66.1 (QE split).
- On proof verification, the best model achieves a Balanced F1 of only 65.1, with a low true-negative rate.
- Detecting critical errors in proofs is the main bottleneck for current models.
Summary
Large language models (LLMs) have achieved remarkable performance on high-school and olympiad-style mathematics, yet their capabilities on advanced mathematics remain poorly understood. Existing benchmarks fall short in both scope and evaluation granularity: they offer limited disciplinary coverage and often rely on final-answer correctness or coarse judgments, leaving the validity of the reasoning process inadequately assessed.
To bridge this gap, the authors introduce AdvancedMathBench, a benchmark suite for evaluating advanced mathematical reasoning:
Key Findings
Original Abstract (excerpt)
> Large language models (LLMs) have achieved remarkable performance on high-school and olympiad-style mathematics, yet their capabilities on advanced mathematics remain poorly understood. Existing benchmarks, however, fall short in both scope and evaluation granularity: they provide limited disciplinary coverage and often rely on final-answer correctness or coarse judgments, leaving the validity of the reasoning process inadequately assessed. To bridge this gap, we introduce AdvancedMathBench, a benchmark suite designed to evaluate advanced mathematical reasoning capabilities. Its core proof-generation benchmark, ProverBench, contains 296 problems spanning undergraduate and doctoral qualifying-exam levels. To provide reliable evaluation of the proofs, we develop a dedicated automatic verific...
---
*Auto-collected on 2026-07-15.*