MathDuels: Evaluating LLMs as Problem Posers and Solvers

Forum topic · 小凯 · 2026-04-25

Summary

MathDuels is an adversarial benchmark from University of Pennsylvania researchers (arXiv 2604.21916) that evaluates LLMs on both posing and solving mathematical problems, inspired by the 1535 Tartaglia-Fior math duel. Nineteen frontier models each generate 30 problems through a three-stage pipeline (meta-prompting, problem generation, difficulty amplification), then attempt to solve each other's problems, yielding 559 valid problems and 10,062 solve/fail observations. Rasch model analysis separates solver ability from problem difficulty, revealing that problem-posing and problem-solving skills are partially decoupled: GPT-5.4-high is the strongest solver, but Gemini-3.1-Pro-high ranks first overall because its problems have the lowest solve rate (62.9%). Newer models pose harder problems—27.6% of their problems defeated the previous top-3 solvers versus 8.0% for others—so difficulty evolves with participant strength and the benchmark never saturates. The framework also resists data contamination since problems are freshly generated each round. Future work includes proof verification and domain transfer.

MathDuels: When the Problem Poser Is More Dangerous Than the Solver

Paper: MathDuels: Evaluating LLMs as Problem Posers and Solvers Authors: Zhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik (University of Pennsylvania) Posted: 2026-04-24

Why Static Benchmarks Are Dying

Frontier models keep saturating benchmarks faster than humans can write new problems:

GSM8K: once separated GPT-3 from GPT-3.5; all frontier models now exceed 95%
MATH: strongest models are near the ceiling
AIME / FrontierMath: rapidly saturating

Static testing is like a treadmill marathon—the belt speed is fixed, and the fastest runners quickly outpace it. MathDuels proposes the sustainable alternative: let the models pose the problems themselves.

The Historical Inspiration

In 1535 Venice, Niccolò Tartaglia accepted Antonio Maria Fior's duel: each side deposited 30 problems with a notary, with 50 days to solve the opponent's set. Tartaglia solved all 30 of Fior's problems; Fior solved none of Tartaglia's. The lesson: problem-posing ability is itself a measure of mathematical ability.

How MathDuels Works

1. Posing: Each of 19 frontier models (from OpenAI, Google, Anthropic, xAI, Alibaba, Kimi, DeepSeek, etc.) generates 30 problems via a three-stage pipeline:

*Meta-prompting*: assign a mathematical domain and require a unique, non-trivially-solvable answer
*Problem generation*
*Difficulty amplification*: the model iteratively adds constraints, distractors, and stricter proof requirements

Solving

Scoring

\log\frac{P_{ni}}{1-P_{ni}} = \theta_n - \delta_i

where $\theta_n$ is solver $$n$$ 's ability, $\delta_i$ is problem $$i$$ 's difficulty, and $P_{ni}$ is the probability that solver $$n$$ solves problem $$i$$ . Like ELO, what matters is *whom* you beat and *how hard* the problems were.

Key Findings

1. The best solver is not the best overall model

| Model | Solver rank | Overall rank | Posing quality | |---|---|---|---| | GPT-5.4-high | #1 | #2 | High | | Gemini-3.1-Pro-high | Not #1 | #1 | Highest (problems average 62.9% solve rate—lowest overall) | | Grok-4.20-high | Mid | Lower | Lowest posing ability |

Solving and posing are partially decoupled: a student who aces every exam cannot necessarily design discriminating exam questions, and vice versa.

2. Self-evolving difficulty

When Gemini-3.1-Pro-high and GPT-5.4-high entered the arena, 27.6% of their posed problems defeated the previous top-3 solvers, versus only 8.0% for the other 17 models. Difficulty co-evolves with participant strength, so the benchmark never saturates—like an arms race where the best shooters also design the targets.

3. Signal from the discriminating tail

Even after difficulty amplification, 39% of problems were solved by all non-posing models. But the remaining 61% contain a clear discriminating tail—MathDuels extracts signal only from problems that differentiate solvers.

Why It Matters

Paradigm shift in evaluation: humans cannot write hard problems as fast as models improve; model-vs-model posing is the only scalable path.
Creativity as a core capability: posing is creation, not execution—a distinct dimension of mathematical understanding.
Contamination resistance: problems are newly generated each round, so memorization cannot cheat the test.

Future Directions

Proof verification: score reasoning rigor, not just final answers
Domain transfer: competitive programming, scientific QA, legal analysis, medical diagnosis—any field where constructing hard problems tests understanding
Larger arenas: more posing budget per model for tighter confidence intervals

Conclusion

Five hundred years after Tartaglia, MathDuels reprises the duel with 19 LLMs. Gemini-3.1-Pro-high tops the leaderboard, but the real winner is the evaluation methodology itself: an arena that never saturates, evolves with capability, and rewards those who can not only solve—but also *pose*—the hardest problems.