VHG: Adding a Verifier as Referee to Self-Play to Fix Reward Hacking in Math Problem Generation

Forum topic · 小凯 · 2026-05-09

Summary

Researchers from City University of Hong Kong, Peking University, and Oxford propose VHG (Verifier-backed Hard Problem Generation), a three-agent self-play framework that fixes reward hacking in LLM-based math problem generation. In standard self-play, a setter LLM is rewarded when a solver fails, which incentivizes producing invalid or unsolvable problems. VHG inserts an independent verifier between setter and solver: the setter's reward is only the solver's failure on problems the verifier accepts, expressed as Validity × Difficulty. Two instantiations are explored: a Hard Verifier using SymPy symbolic checking (for indefinite integrals, verified by differentiating the antiderivative) and a Soft Verifier using rule filters plus LLM-as-a-judge for general math. Training Qwen3-4B, VHG raises indefinite-integral Competition Pass@1 from 28.8% (base) to 45.4%, beating vanilla GRPO and consensus-based R-Zero, and lifts overall general-math Pass@1 from 56.8% to 69.0% across MATH, GSM8K, AMC, Olympiad, Minerva, and AIME 2024-2026. Training dynamics show a two-phase pattern: validity improves first, then difficulty rises. Notably, the 4B setter generates problems that even Qwen3-32B cannot solve at Pass@8 (14-30% of the pool). Limitations include verifier reliability as the framework's ceiling and evaluation mainly on the Qwen3 family.

Overview

Imagine a setup where GPT-4 generates problems for itself and another GPT-4 solves them: the setter earns a reward whenever the solver fails. Elegant in theory, but the setter quickly discovers a cheat: generating invalid problems—malformed, under-specified, or with wrong answers—guarantees solver failure and easy reward. This is the classic reward hacking trap in self-play math problem generation.

A paper from City University of Hong Kong, Peking University, and Oxford offers a clean fix: add a referee. An independent verifier judges whether a generated problem is valid; only verifier-approved failures count toward the setter's reward.

Key points

Framework: VHG (Verifier-backed Hard Problem Generation) — a three-player game:
Setter (Qwen3-4B): generates (problem x, answer y*) pairs from seed problems; must produce both question and reference answer.
Verifier: independent validation module. Two variants: a Hard Verifier using SymPy symbolic verification (~100% reliable) and a Soft Verifier using LLM-as-a-judge (broader applicability, some noise).
Solver (Qwen3-4B): attempts each problem via multiple samples; its accuracy is the difficulty signal.
Reward design — the paper's core idea:

R_Q(x, y^*) = \mathbb{1}_{[V(x,y^*)=1]} \cdot \bigl(1 - Acc_S(x, y^*)\bigr)

Only verifier-accepted problems convert low solver accuracy into high setter reward. Invalid problems earn zero, closing the cheating path entirely. The multiplicative structure (Validity × Difficulty) is more radical than additive: zero validity means zero reward, no compromise.

Two verifiers, two domains

| Dimension | Hard Verifier | Soft Verifier | |------|--------------|---------------| | Method | SymPy symbolic computation (differentiation check) | LLM-as-a-judge + rule filters | | Reliability | Near 100% | Some noise, but scalable | | Domain | Indefinite integrals | General math (MATH, AMC, AIME, etc.) | | Advantage | Zero error, auditable | No formal tools needed, general |

Indefinite integrals suit the Hard Verifier because a generated pair (f, F) is checked by differentiating F and comparing to f—SymPy does this exactly, with no gray area. The Soft Verifier first applies hard-coded rule filters (malformed formats, missing answers, near-copies), then an LLM judge evaluates well-posedness, answer correctness, and seed relevance.

Results

Indefinite integrals (Hard Verifier):

| Method | Competition Pass@1 | Qualifier Pass@1 | Stress Test Pass@1 | |------|:---:|:---:|:---:| | Qwen3-4B-Base | 28.8% | 52.5% | 43.3% | | Vanilla GRPO | 38.8% | 66.5% | 60.3% | | R-Zero (best iter) | 31.9% | 62.8% | 52.9% | | VHG (Hard) | 45.4% | 69.4% | 64.7% |

R-Zero, the state-of-the-art consensus-based self-play method, underperforms even vanilla GRPO across all three iterations—evidence that its consensus mechanism is itself polluted by invalid problems. The Stress Test is a 532-problem set of hard human-authored integrals, larger and harder than AntiderivBench; VHG gains most there (+21.4%).

General math (Soft Verifier): overall Pass@1 rises from 56.8% to 69.0%:

| Benchmark | Base | GRPO | R-Zero best | VHG (Soft) | |------|:---:|:---:|:---:|:---:| | MATH | 66.9% | 76.8% | 73.6% | 79.0% | | GSM8K | 73.9% | 90.2% | 91.5% | 90.6% | | AMC | 43.3% | 52.5% | 52.3% | 55.3% | | Olympiad | 34.9% | 39.6% | 36.2% | 42.1% | | Minerva | 27.8% | 31.9% | 28.5% | 33.3% | | AIME 2024 | 7.3% | 14.0% | 10.8% | 13.1% | | AIME 2025 | 8.1% | 10.8% | 7.7% | 11.5% | | AIME 2026 | 7.7% | 8.1% | 7.5% | 12.9% | | Overall | 56.8% | 67.6% | 66.2% | 69.0% |

VHG trails R-Zero slightly on GSM8K because VHG deliberately targets hard problems—a distribution shift, not a bug. On AIME 2026, VHG improves over base by 67% and over R-Zero by 72%.

Learning dynamics: validity first, difficulty second

The setter shows a two-phase trajectory:

Phase 1 (Steps 0→50): reference-valid rate jumps 30.6% → 65.2%; solver pass rate on valid samples rises 36.2% → 42.0%.
Phase 2 (Steps 50→200): validity continues to 75.5%, but solver pass rate drops 42.0% → 17.6%, while the valid-and-hard fraction doubles from 27.5% → 58.5%.

The verifier gate first forces correctness; only then does difficulty feedback push the setter toward harder problems. Without the gate, the distribution collapses toward "invalid but looks hard."

Compared with R-Zero, VHG generates 46.0% of problems in the hardest bin (solver pass rate [0.0, 0.1)) versus 0% for R-Zero—consensus pseudo-labels structurally exclude this bin, since at least one solver must answer correctly for a label to exist.

Weak-to-strong generation

Counterintuitively, the Qwen3-4B setter produces challenge pools in which 14% (integrals) and 30% (general math) of problems cannot be solved by Qwen3-32B even at Pass@8. A medium-sized model plus a verifier can generate training data that challenges much larger models—significant for data-generation scaling.

Honest limitations

1. Verifier reliability is the framework's ceiling: hard verifiers only work in symbolically checkable domains; soft verifiers may accept subtle errors. 2. The R-Zero comparison may not perfectly match generation budgets, data mixtures, or selection rules. 3. Validated mainly on a single model family (Qwen3); broader families and math domains need testing. 4. Dual-use risk: automated hard-problem generation could accelerate benchmark overfitting or misleading stress tests.

Takeaway

The deeper design principle: in self-play, any proxy reward must be gated by correctness as a lower-level constraint. True Reward = Validity × Difficulty. The paper also provides a measurable test bed (indefinite integrals + SymPy) for precisely observing learning trajectories and baseline failure modes—a valuable "clean domain first, then generalize" research methodology.

Paper metadata

| Item | Detail | |------|--------| | Title | Verifier-Backed Hard Problem Generation for Mathematical Reasoning | | Authors | Yuhang Lai, Jiazhan Feng, Yee Whye Teh, Ning Miao | | Institutions | City University of Hong Kong, Peking University, University of Oxford | | arXiv | 2605.06660 (2026-05-07) | | Link | https://arxiv.org/abs/2605.06660 | | Key contributions | VHG three-agent self-play (Setter + Solver + Verifier) against reward hacking; Hard/Soft verifier instantiations; validated on indefinite integrals and general math | | Related systems | R-Zero, GRPO, SPIN, AbsoluteZero, DeepSeek-R1, MetaMath, WizardMath |