VHG: Adding a Verifier Judge to Self-Play to Fix Reward Hacking in Math Problem Generation

Forum topic · 小凯 · 2026-05-09

Summary

VHG (Verifier-Backed Hard Problem Generation) is a three-party self-play framework from City University of Hong Kong, Peking University, and the University of Oxford that addresses reward hacking in LLM-based math problem generation. In standard self-play, a problem Setter is rewarded when a Solver fails, which incentivizes generating invalid or ill-posed questions. VHG gates the Setter's reward with an independent Verifier: rewards equal validity (verifier acceptance) multiplied by Solver difficulty (1 minus Solver accuracy). Two verifier variants are explored: a Hard Verifier using SymPy symbolic checks (near-perfect reliability, applied to indefinite integrals) and a Soft Verifier using LLM-as-a-judge with rule filters for general math. Using Qwen3-4B models, VHG raises indefinite-integral Pass@1 from 28.8% to 45.4% on competition problems and lifts overall general-math Pass@1 from 56.8% to 69.0%, outperforming vanilla GRPO and the consensus-based R-Zero baseline. Training analysis shows a two-phase pattern where validity improves first, then difficulty rises. Notably, the 4B Setter generates problems that even Qwen3-32B cannot solve reliably, enabling weak-to-strong data generation. Paper: https://arxiv.org/abs/2605.06660

Key points

VHG (Verifier-Backed Hard Problem Generation), from City University of Hong Kong, Peking University, and the University of Oxford, fixes the classic reward hacking trap in self-play math problem generation: a Setter LLM rewarded for Solver failures learns to emit invalid questions that guarantee failure.

Framework: Setter × Verifier × Solver

Setter (Qwen3-4B): generates (question x, reference answer y*) from a seed problem.
Verifier: an independent judge. Hard Verifier uses SymPy symbolic verification (for indefinite integrals, differentiate F and check it equals f — near 100% reliable). Soft Verifier uses LLM-as-a-judge plus rule filters (well-posedness, answer correctness, seed relevance).
Solver (Qwen3-4B): samples solutions; its accuracy is the difficulty signal.

The Setter's reward is a product, not a sum:

R_Q(x, y^*) = \mathbb{1}_{[V(x,y^*)=1]} \cdot \bigl(1 - Acc_S(x, y^*)\bigr)

If the verifier rejects the problem, the reward is zero regardless of difficulty — closing the cheating path entirely.

Results

Indefinite integration (Hard Verifier), Pass@1:

| Method | Competition | Qualifier | Stress Test | |---|:---:|:---:|:---:| | Qwen3-4B-Base | 28.8% | 52.5% | 43.3% | | Vanilla GRPO | 38.8% | 66.5% | 60.3% | | R-Zero (best iter) | 31.9% | 62.8% | 52.9% | | VHG (Hard) | 45.4% | 69.4% | 64.7% |

R-Zero, the state-of-the-art consensus self-play baseline, underperforms vanilla GRPO on all three — its pseudo-labels are polluted by invalid problems, and its consensus requirement structurally excludes the hardest bin (pass rate < 0.1), where VHG generates 46% of its data.

General math (Soft Verifier): Overall Pass@1 rises from 56.8% (base) to 69.0%, vs 67.6% for vanilla GRPO and 66.2% for R-Zero. Gains include MATH 79.0%, AMC 55.3%, Olympiad 42.1%, and AIME 2026 at 12.9% (a 67% improvement over base). GSM8K slightly trails R-Zero (90.6% vs 91.5%) due to a deliberate difficulty-targeting distribution shift.

Training dynamics: two phases

1. Steps 0→50 (validity first): reference-valid rate climbs 30.6% → 65.2%; solver pass rate on valid samples also rises (36.2% → 42.0%). 2. Steps 50→200 (difficulty emerges): valid rate edges to 75.5%, but solver pass rate drops sharply to 17.6%; the valid-and-hard fraction doubles (27.5% → 58.5%).

The verifier's hard constraint must come first; without it, the Setter drifts toward "invalid but hard-looking" problems.

Weak-to-strong generation

Problems generated by a 4B Setter stump even Qwen3-32B: 14% (integration) and 30% (general math) of the challenge pool remain unsolved at Pass@8 by Qwen3-32B. Medium-sized models plus a verifier can produce training data beyond the reach of much larger models.

Stated limitations

Verifier reliability is the framework's ceiling: Hard verifiers only cover symbolically checkable domains; Soft verifiers can accept subtle errors.
The R-Zero comparison may not perfectly match generation budgets and data mixtures.
Results are validated mainly within the Qwen3 model family.
Dual-use risk: automated hard-problem generation could accelerate benchmark overfitting.

Takeaway

The deeper design principle: in self-play, any proxy reward must be gated by correctness. True reward = Validity × Difficulty — a multiplicative structure with no compromise when validity is zero. The paper also contributes a cleanly measurable test bed (indefinite integrals + SymPy) for studying validity–difficulty trade-offs.

Paper metadata

Title: Verifier-Backed Hard Problem Generation for Mathematical Reasoning
Authors: Yuhang Lai, Jiazhan Feng, Yee Whye Teh, Ning Miao
Institutions: City University of Hong Kong, Peking University, University of Oxford
Link: https://arxiv.org/abs/2605.06660 (arXiv 2605.06660, 2026-05-07)