AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Reasoning in LLMs

Forum topic · 小凯 · 2026-07-15

Summary

AdvancedMathBench (arXiv: 2607.11849) is a benchmark suite for evaluating large language models' advanced mathematical reasoning beyond high-school and olympiad-level problems. Its proof-generation component, ProverBench, contains 296 problems at undergraduate and doctoral qualifying-exam difficulty. Because final-answer checking is insufficient for proofs, the authors built an automatic verification pipeline trained on large-scale expert annotations that outputs correctness verdicts and fine-grained assessments of proof errors, achieving high agreement with human experts on held-out proof traces. A second component, VerifierBench, pairs 888 model-generated proof traces with expert ground truth to test whether models can correctly judge proof validity and provide sound verification rationale. Results show the benchmark remains challenging for frontier models: the best proof generator, GPT-5.5-xhigh, scores only 75.8 and 66.1 on the UGD and QE splits, and the best proof verifier reaches a Balanced F1 of just 65.1, with low true-negative rates indicating that detecting critical errors is the main bottleneck.

AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Reasoning

Field: NLP Authors: Lingkai Kong, Zijian Wu, Yuzhe Gu, Haiteng Zhao, Wenyong Huang Released: 2026-07-13 arXiv: 2607.11849

Abstract (translated)

Large language models (LLMs) have achieved remarkable performance on high-school and olympiad-style mathematics, yet their capabilities on advanced mathematics remain poorly understood. Existing benchmarks fall short in both scope and evaluation granularity: they offer limited disciplinary coverage and often rely on final-answer correctness or coarse judgments, leaving the validity of the reasoning process inadequately assessed.

To bridge this gap, the authors introduce AdvancedMathBench, a benchmark suite for evaluating advanced mathematical reasoning:

ProverBench — the core proof-generation benchmark, containing 296 problems spanning undergraduate and doctoral qualifying-exam levels.
Automatic verification pipeline — trained on large-scale expert annotations, it outputs correctness verdicts and fine-grained assessments of proof errors, showing high agreement with human experts on held-out proof traces.
VerifierBench — 888 model-generated proof traces paired with expert ground truth, evaluating whether models can correctly judge proof validity and provide sound verification rationale.

Key Findings

AdvancedMathBench remains challenging for frontier models.
Best proof-generation model GPT-5.5-xhigh achieves only 75.8 (UGD split) and 66.1 (QE split).
Best proof-verification model reaches a Balanced F1 of 65.1, with low true-negative rates.
Detecting critical errors in proofs is the main bottleneck for current models.

---

*Auto-collected on 2026-07-15.*