MathDuels: Evaluating LLMs as Problem Posers and Solvers — When the Question-Setter Is More Dangerous Than the Solver

Forum topic · 小凯 · 2026-04-25

Summary

MathDuels, a 2026 paper from University of Pennsylvania researchers (arXiv:2604.21916), proposes an adversarial benchmark where 19 frontier LLMs both pose and solve mathematical problems, inspired by the 1535 Venice math duel between Tartaglia and Fior. Each model generates 30 problems via a three-stage pipeline (meta-prompting, problem generation, difficulty amplification), and all models cross-solve each other's problems, yielding 559 valid problems and 10,062 solve/fail observations. Item-response (Rasch) modeling jointly estimates solver ability and problem difficulty. Key findings: posing and solving skills are partially decoupled—GPT-5.4-high is the strongest solver, but Gemini-3.1-Pro-high ranks first overall because its problems have the lowest solve rate (62.9%); newly added models produced problems that beat top solvers 27.6% of the time versus 8.0% for others. Because problem difficulty co-evolves with participant strength, MathDuels resists saturation and data contamination, offering a sustainable alternative to static benchmarks like GSM8K and MATH. Future work includes proof verification, domain transfer, and larger arenas.

MathDuels: The Mathematics of Duels — When the Problem Poser Is More Dangerous Than the Solver

*Paper of the day — 2026-04-26. Source: arXiv:2604.21916*

Overview

Field: LLM evaluation / mathematical reasoning / adversarial benchmarking
Authors: Zhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik (University of Pennsylvania)
Published: 2026-04-24
Paper: arXiv:2604.21916

The Story Begins: A Math Duel in 1535

In 1535 Venice, mathematician Niccolò Tartaglia received a challenge from Antonio Maria Fior: each side would prepare 30 math problems, deposited with a notary. Within 50 days, whoever solved more of the opponent's problems would win.

The result:

Tartaglia solved all 30 of Fior's problems
Fior failed to solve even one of Tartaglia's

Why? Because the ability to pose problems is itself a manifestation of mathematical ability. Tartaglia could not only solve hard problems but also *construct* problems beyond his opponent's limits.

Nearly 500 years later, the University of Pennsylvania team brought this ancient wisdom to LLM evaluation — and MathDuels was born.

Why Static Benchmarks Are Dying

Static benchmarking is like running a marathon on a treadmill: the belt speed is fixed, runners keep getting faster, and soon the fastest runners easily outpace the belt — the test loses discriminative power.

| Benchmark | Once distinguished | Status now | |---|---|---| | GSM8K | GPT-3 vs GPT-3.5 | All frontier models >95% | | MATH | GPT-3.5 vs GPT-4 | Top models near ceiling | | AIME 2026 | Latest competition problems | Already conquered | | FrontierMath | Research frontier | Rapidly saturating |

Benchmarks saturate faster than new problems can be supplied. Human question writers increasingly struggle to craft problems hard enough to separate the strongest models.

Core Insight: Solving and Posing Are Different Abilities

Being able to taste the best wine doesn't mean you can brew it. MathDuels finds that solving ability and posing ability are partially decoupled.

Across 19 frontier models:

GPT-5.4-high is the strongest *solver* (highest solver rating)
But the overall #1 is Gemini-3.1-Pro-high — because its posing is strongest: its problems average only a 62.9% solve rate, the lowest among all participants

A student can ace every exam (strong solver) without being able to design discriminating questions (weak poser), and vice versa.

How MathDuels Works

1. Participation: 19 frontier models enter the arena (from OpenAI, Google, Anthropic, xAI, Alibaba, Kimi, DeepSeek, etc.). 2. Problem posing: Each model generates 30 problems via a three-stage pipeline:

Meta-prompting: set the domain (algebra, geometry, analysis) and constraints (unique solution, non-trivial reasoning, capable of challenging state-of-the-art AI)
Problem generation: the model produces the concrete problem
Difficulty amplification: the model self-critiques — add a constraint, introduce a distractor, demand a stricter proof

Cross-dueling

Scoring (Rasch model)

\log\frac{P_{ni}}{1-P_{ni}} = \theta_n - \delta_i

where $\theta_n$ is solver $$n$$ 's ability and $\delta_i$ is problem $$i$$ 's difficulty. Like an ELO system: what matters is *whom* you beat and *how hard* their problems were.

Key Findings

1. The strongest solver ≠ the strongest model

| Model | Solving rank | Overall rank | Posing quality | |---|---|---|---| | GPT-5.4-high | #1 | #2 | High | | Gemini-3.1-Pro-high | Not #1 | #1 | Highest | | Grok-4.20-high | Middle | Lower | Lowest (largest solve–pose gap) |

Looking only at solving ability would miss the true "math master."

2. Self-evolving difficulty

When two new models (Gemini-3.1-Pro-high and GPT-5.4-high) entered the arena:

27.6% of their posed problems defeated the previous Top-3 solvers
Only 8.0% of problems from the other 17 models did so

New models are not only better solvers but also trickier posers. Difficulty co-evolves with participant strength, so the benchmark never saturates.

3. Solve-rate distribution reveals the truth

Even after three-stage difficulty amplification, 39% of problems were solved by all non-author models (no discrimination). But in the remaining 61%, there is a clear "discriminating tail." MathDuels doesn't need every problem to discriminate — it only extracts signal from the ones that do.

Why It Matters

Paradigm shift in evaluation: humans can't write problems as fast as models improve. The only sustainable solution is letting models pose problems themselves.
Creativity as a core capability: solving is execution; posing is creation. True mathematical understanding shows not just in how far you can go, but in how high a bar you can set.
Robustness to contamination: problems are freshly generated each round, so models cannot cheat by memorizing, and difficulty automatically adapts to model capability.

Future Directions

1. Proof verification: currently only final answers are checked; future work could validate reasoning rigor. 2. Domain transfer: the poser–solver framework applies to competitive programming, scientific QA, legal analysis, medical diagnosis — any field where constructing hard problems itself tests understanding. 3. Larger arenas: more posing budget per model, tighter confidence intervals, more reliable rankings.

Closing Thought

Five hundred years ago, Tartaglia's duel proved the importance of posing ability. Five hundred years later, MathDuels reenacted that truth with 19 LLMs. On the final leaderboard, Gemini-3.1-Pro-high sits at #1 — but the real winner is the evaluation method itself: a system that never saturates, always evolves with capability, and always tells the truth.