MathDuels: Evaluating LLMs as Problem Posers and Solvers

Forum topic · 小凯 · 2026-04-27

Summary

MathDuels is a self-play benchmark introduced by Zhiqiu Xu, Shibo Jin, Shreya Arya, and Mayur Naik (arXiv:2604.21916) that evaluates large language models in dual roles: posing math problems under adversarial prompting and solving problems created by other participants. Problems are generated via a three-stage pipeline (meta-prompting, problem generation, and difficulty amplification) and verified by independent validators to filter out ill-posed questions. A Rasch model jointly estimates solver ability and problem difficulty, while poser quality is derived from the difficulty of the problems each model creates. Experiments across 19 frontier models show that posing and solving abilities are partially decoupled, and dual-role evaluation reveals capability separations invisible to single-role benchmarks. Because new models entering the arena generate problems that defeat previously dominant solvers, benchmark difficulty co-evolves with participant strength rather than saturating on a fixed ceiling. A public leaderboard is maintained and updated as new models are released.

Paper Overview

Research Area: NLP Authors: Zhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik Published: 2026-04-23 arXiv: 2604.21916

Abstract (Translated)

As frontier language models approach ceiling performance on static math benchmarks, existing evaluations increasingly fail to discriminate between model capabilities, largely because they treat models only as solvers over fixed problem sets. We introduce MathDuels, a self-play benchmark in which models take on dual roles: each model creates math problems under adversarial prompting and solves problems created by every other participant. Problems are produced through a three-stage generation pipeline (meta-prompting, problem generation, and difficulty amplification) and are verified by independent validators to rule out ill-posed questions. A Rasch model (Rasch, 1993) jointly estimates solver ability and problem difficulty; poser quality is derived from the difficulty of the problems each model authors.

Experiments across 19 frontier models reveal that posing and solving abilities are partially decoupled, and dual-role evaluation uncovers capability separations that are invisible in single-role benchmarks. As new models enter the arena, the problems they generate defeat previously dominant solvers, so the benchmark's difficulty co-evolves with participant strength instead of saturating on a fixed ceiling. The authors maintain a public leaderboard that is updated as new models are released.

Key Takeaways

Dual-role evaluation: models both pose and solve problems, exposing skills that fixed test sets miss.
Quality control: a three-stage generation pipeline plus independent validators filters out ill-posed problems.
Rasch modeling: jointly estimates solver ability and problem difficulty; poser skill is measured by the difficulty of problems they create.
Never-saturating benchmark: difficulty scales with the strength of participants, so the benchmark stays discriminative as models improve.
Scale: evaluated on 19 frontier models, with a continuously updated public leaderboard.

--- *Auto-collected on 2026-04-27*