MathDuels: Evaluating LLMs as Problem Posers and Solvers

Forum topic · 小凯 · 2026-04-25

Summary

MathDuels is a self-play benchmark that evaluates large language models in dual roles: as adversarial problem authors and as solvers of problems written by every other participant. The framework addresses the saturation of static math benchmarks, where frontier models now approach ceiling performance. Problems are generated via a three-stage pipeline (meta-prompting, problem generation, and difficulty amplification) and filtered by an independent verifier that removes ill-posed questions. A Rasch model jointly estimates solver abilities and problem difficulties, while each model's authoring quality is derived from the difficulty of the problems it creates. Experiments across 19 frontier models show that authoring and solving abilities are only partially decoupled, and the dual-role evaluation reveals capability gaps invisible to single-role benchmarks. Because newly released models contribute harder problems that can defeat previously dominant solvers, benchmark difficulty co-evolves with participant strength rather than saturating at a fixed ceiling. A public leaderboard is maintained and updated with new model releases. Paper: arXiv 2604.21937 by Zhiqiu Xu, Shibo Jin, and Shreya Arya.

Paper Overview

Field: NLP Authors: Zhiqiu Xu, Shibo Jin, Shreya Arya Published: 2026-04-23 arXiv: 2604.21937

Motivation

As frontier language models attain near-ceiling performance on static mathematical benchmarks, existing evaluations are increasingly unable to differentiate model capabilities, largely because they cast models solely as solvers of fixed problem sets.

Approach

MathDuels is a self-play benchmark in which models occupy dual roles:

Authoring: each model writes math problems under adversarial prompting.
Solving: each model solves problems authored by every other participant.

Problems are produced through a three-stage generation pipeline:

1. Meta-prompting 2. Problem generation 3. Difficulty amplification

An independent verifier validates the problems and excludes ill-posed questions.

Evaluation Method

A Rasch model jointly estimates solver abilities and problem difficulties.
Author quality is derived from the difficulties of each model's authored problems.

Key Findings

Experiments cover 19 frontier models.
Authoring and solving abilities are partially decoupled.
Dual-role evaluation reveals capability differences invisible to single-role benchmarks.
As new models enter the arena, their authored problems defeat previously dominant solvers, so benchmark difficulty co-evolves with participant strength instead of saturating at a fixed ceiling.

A public leaderboard is hosted and updated as new models are released.

---

*Auto-collected on 2026-04-25*