Verifier-Backed Hard Problem Generation (VHG): A Verifier-Gated Three-Way Self-Play Framework for Mathematical Reasoning

Forum topic · 小凯 · 2026-05-09

Summary

VHG (Verifier-Backed Hard Problem Generation) is a three-way self-play framework in which a Setter LLM generates new math problems with reference answers, an independent Verifier gates their validity, and a Solver LLM's empirical accuracy serves as the difficulty signal. The Setter reward is restructured as validity-gated difficulty (Validity × Difficulty), eliminating the reward-hacking incentive present in consensus-based self-play methods like R-Zero. Two verifier instantiations are tested: a hard symbolic verifier (SymPy-based) for indefinite integrals, and a soft LLM-as-a-judge verifier for general math. Using Qwen3-4B-Base with GRPO-style RL, VHG raises Pass@1 on a human-authored hard-integral stress test by +21.4% over the base model and improves overall general-math Pass@1 from 56.8% to 69.0%. Generated 4B-model problems challenge Qwen3-32B (Pass@8 unsolved up to 30%). Learning dynamics show a two-phase pattern: validity first, then difficulty. Paper: https://arxiv.org/abs/2605.06660

Overview

VHG (Verifier-Backed Hard Problem Generation) is a three-way self-play framework for generating difficult, valid math problems to train LLM reasoning. It addresses reward hacking in existing self-play generation methods (e.g., R-Zero) by embedding an independent verifier directly into the reward function.

Paper: Verifier-Backed Hard Problem Generation for Mathematical Reasoning
Authors: Yuhang Lai, Jiazhan Feng, Yee Whye Teh, Ning Miao (City University of Hong Kong; Peking University; University of Oxford)
arXiv: https://arxiv.org/abs/2605.06660

Key Idea: Validity-Gated Difficulty Reward

Three modules: Setter $Q_\theta$ generates problem–answer pairs $$(x, y^*)$$ conditioned on seeds; a Verifier $$V$$ performs acceptance checks; Solver $S_\phi$ produces $$K$$ solutions whose accuracy is the difficulty signal.

Setter reward:

R_Q(x, y^*) = \mathbb{1}_{[V(x,y^*)=1]} \cdot (1 - Acc_S(x, y^*))

The multiplicative structure ensures that when validity is zero, the difficulty reward is fully gated out. Unlike consensus-backed rewards (R-Zero), invalid problems cannot pollute training via majority votes. The Solver is trained only on the verifier-accepted pool $\mathcal{D}_V$ with $$R_S = Acc_S$$ .

Two Verifier Instantiations

Hard verifier (indefinite integrals): SymPy-based check — the candidate antiderivative must differentiate back to the integrand with well-formed formatting. Near-100% reliability, fully auditable, but limited to symbolically checkable domains.
Soft verifier (general math): Hard-coded rule filters (format, near-copy, degenerate answers) plus an LLM judge (GPT-5.4) assessing well-posedness, answer correctness, and seed relevance. Trades absolute reliability for domain generality.

Experimental Setup

Base model: Qwen3-4B-Base (shared Setter and Solver)
Training: verl + GRPO-style RL, 8 GPUs, ~60 hours per full cycle
Evaluation sampling: 64 samples/problem (integrals), 16/problem (general math)

Main Results

Indefinite Integrals (Pass@1)

| Method | Competition | Qualifier | Stress Test (532 human-authored hard problems) | |---|---|---|---| | Qwen3-4B-Base | 28.8% | 52.5% | 43.3% | | Vanilla GRPO | 38.8% | 66.5% | 60.3% | | R-Zero (best iter) | 31.9% | 62.8% | 52.9% | | VHG (Hard) | 45.4% | 69.4% | 64.7% (+21.4%) |

Notably, R-Zero underperforms even Vanilla GRPO across iterations, consistent with its reward-hacking/consensus-pollution problem.

General Math (Overall Pass@1)

| Method | MATH | GSM8K | AIME24 | AIME25 | AIME26 | Overall | |---|---|---|---|---|---|---| | Base | 66.9 | 73.9 | 7.3 | 8.1 | 7.7 | 56.8 | | Vanilla GRPO | 76.8 | 90.2 | 14.0 | 10.8 | 8.1 | 67.6 | | R-Zero (best) | 73.6 | 91.5 | 10.8 | 7.7 | 7.5 | 66.2 | | VHG (Soft) | 79.0 | 90.6 | 13.1 | 11.5 | 12.9 | 69.0 |

Overall improves 56.8% → 69.0% (+21.5% relative). On AIME 2026, VHG reaches 12.9% (+67% over base, +72% over R-Zero). The slight GSM8K dip vs. R-Zero reflects the intended focus on hard problems.

Difficulty and Mechanism Analysis

Verifier-valid generated pools concentrate in low Pass@1 regions. Even Qwen3-32B leaves 14% (integrals) and 30% (general math) of the zero-Pass@1 problem pool unsolved at Pass@8 — a weak 4B model generates data that challenges strong models.
Two-phase Setter learning: early training is dominated by validity learning (valid rate 30.6% → 65.2%); once validity is established, difficulty feedback dominates — solver pass rate on valid samples drops 42.0% → 17.6%, and the valid-and-hard ratio doubles (27.5% → 58.5%).
Distribution gap vs. R-Zero: consensus pseudo-labels require at least one solver to answer correctly, structurally excluding the hardest bin. VHG places 46% of problems in the [0.0, 0.1) pass-rate bin (vs. 0% for R-Zero Iter.2) while maintaining higher validity across all bins.
Data quality: integral pipeline yields 9,077 globally novel verifier-matched problems (4.6% seed-copy rate); general-math pipeline filters 400,000 candidates down to 16,536 training rows (44.9% judge accept rate, ~4.1% yield).

Relation to Prior Work

MetaMath / WizardMath: rewrite seed problems; problem source is fixed.
R-Zero: consensus self-play without an independent verifier; vulnerable to reward hacking.
AbsoluteZero: self-proposed problems verified by code execution; targets code/formal settings, not natural-language math.
DeepSeek-R1: verifiable rewards over external prompts, not an independent verifier gate.
MathSmith: predefined structural constraints, not a learnable verifier.

VHG's distinct contribution is embedding the verifier inside the Setter's reward itself, making difficulty reward conditional on validity rather than applying post-hoc filtering.

Limitations

Hard verifiers apply only to symbolically checkable domains.
Soft verifiers are noisy and may accept subtle errors (mitigations: stronger/multi-judge voting, human-in-the-loop).
Experiments use a single model family (Qwen3).
Automated hard-problem generation risks benchmark contamination.
Large funnel loss in the general-math pipeline (400K → 16.5K).

Conclusion

VHG shows that any difficulty-targeted proxy reward in self-play must be gated by a more fundamental correctness constraint. With $R = \text{Validity} \times \text{Difficulty}$ , a 4B model can generate valid problems that challenge 32B models, and the resulting data beats both consensus self-play (R-Zero) and vanilla GRPO across integral and general-math benchmarks.