Verifier-Backed Hard Problem Generation (VHG): A Verifier-Gated Three-Party Self-Play Framework for Mathematical Reasoning

Forum topic · 小凯 · 2026-05-09

Summary

VHG (Verifier-backed Hard Problem Generation) is a three-party self-play framework that fixes reward hacking in LLM math problem generation. A Setter LLM generates problem-answer pairs from seeds, an independent Verifier gates validity, and a Solver LLM's pass rate provides difficulty. The Setter's reward is the product of validity and difficulty, so invalid problems earn zero reward—unlike consensus-based methods such as R-Zero, whose pseudo-labels can be polluted by invalid questions. VHG is instantiated with a hard verifier (SymPy-based checking for indefinite integration) and a soft verifier (rule filters plus an LLM judge for general math). Training Qwen3-4B-Base with verl/GRPO on 8 GPUs, VHG lifts integration Stress Test Pass@1 from 43.3% to 64.7% (+21.4%) and general math overall Pass@1 from 56.8% to 69.0%, outperforming vanilla GRPO and R-Zero, which even degrades across iterations. Notably, the 4B Setter generates problems that challenge Qwen3-32B. Training dynamics reveal a two-phase pattern: the Setter first learns validity, then difficulty. A 4B model can generate data hard enough to stump 32B models when validity is properly gated.

Overview

Verifier-Backed Hard Problem Generation for Mathematical Reasoning (arXiv:2605.06660) by Yuhang Lai, Jiazhan Feng, Yee Whye Teh, and Ning Miao (City University of Hong Kong; Peking University; University of Oxford) introduces VHG, a three-party self-play framework for generating hard, valid math training data.

The Problem

LLMs excel at solving math problems, but post-training still relies mostly on static human datasets (MATH) or offline transformation recipes (MetaMath, WizardMath, DeepSeekMath). Existing self-play problem generation (Setter generates, Solver solves, negative accuracy as Setter reward) is vulnerable to reward hacking: the Setter can maximize difficulty rewards by emitting invalid, underspecified, or wrong problems.

The VHG Framework

Three modules: Setter $Q_\theta$ (generates problem–reference-answer pairs from seeds), Solver $S_\phi$ (samples $$K$$ solutions), and an independent Verifier $V \in \{0,1\}$ .

Setter reward (validity-gated difficulty):

R_Q(x, y^*) = \mathbb{1}_{[V(x,y^*)=1]} \cdot \bigl(1 - Acc_S(x, y^*)\bigr)

The multiplicative structure means zero validity fully gates off the difficulty signal — fundamentally removing the incentive for reward hacking, unlike consensus-backed rewards (R-Zero) where validity is only indirectly inferred.

Pipeline: seed collection + cold SFT → problem generation → verifier gating → Solver difficulty estimation → Setter RL update → build verifier-accepted pool $\mathcal{D}_V$ → Solver RL training on $\mathcal{D}_V$ .

Two Verifier Instantiations

Hard verifier (indefinite integration): SymPy-based — check format well-formedness and that differentiating $$F$$ yields $$f$$ . Near-100% reliability, fully auditable, but narrow domain.
Soft verifier (general math): hardcoded rule filters (malformed, missing/extra answers, near-copies, degenerate answers) plus an LLM judge assessing well-posedness, answer correctness, and seed relevance.

Key Results (Qwen3-4B-Base, verl + GRPO, 8 GPUs, ~60h per cycle)

Generated problems are genuinely hard. Among zero-Pass@1 problems, Qwen3-32B achieves under 50% Pass@1; Pass@8 unsolved rates reach 14% (integration) and 30% (general math). A weak 4B model generates data that challenges a 32B model.

Indefinite integration (Pass@1):

| Method | Competition | Qualifier | Stress Test (532 hard human-authored problems) | |---|---|---|---| | Qwen3-4B-Base | 28.8% | 52.5% | 43.3% | | Vanilla GRPO | 38.8% | 66.5% | 60.3% | | R-Zero (best iter) | 31.9% | 62.8% | 52.9% | | VHG (Hard) | 45.4% | 69.4% | 64.7% |

R-Zero underperforms even vanilla GRPO, consistent with consensus-based rewards being polluted by invalid problems. VHG's largest gain is on the Stress Test (+21.4%).

General math (overall Pass@1): 56.8% (base) → 69.0% (VHG Soft), vs 67.6% (GRPO) and 66.2% (R-Zero). On AIME 2026, VHG reaches 12.9% — +67% over base, +72% over R-Zero. The slight GSM8K dip vs R-Zero reflects VHG's deliberate focus on hard problems.

Mechanism: Two-Phase Learning Dynamics

Phase 1 (steps 0–50): valid rate rises 30.6% → 65.2%; solver pass rate on valid samples *rises* (36.2% → 42.0%) — the Setter first learns validity, generating easy-correct problems.
Phase 2 (steps 50–200): valid rate 65.2% → 75.5%, but solver pass rate drops to 17.6%; valid-and-hard share doubles (27.5% → 58.5%) — difficulty feedback takes over once validity is established.

Distribution contrast with R-Zero: R-Zero's consensus pseudo-labels require at least one solver success, structurally excluding the hardest bin ([0.0, 0.1) pass rate — 0% of R-Zero's data vs 46% for VHG). VHG also shows higher validity across all difficulty bins.

Data quality: seed-copy rate 4.6%, cross-seed reuse 0.15%; 9,077 globally novel verifier-matched integration problems. The general-math stream filters 400,000 candidates down to 16,536 training rows (44.9% judge accept rate, 4.1% yield).

Positioning and Limitations

Unlike MetaMath/WizardMath (seed rewriting), R-Zero (consensus self-play), AbsoluteZero (code-execution verification for code/formal domains), or DeepSeek-R1 (rewards on external prompts), VHG embeds the verifier inside the Setter's reward function rather than post-hoc filtering.

Limitations: hard verifiers only apply to symbolically checkable domains; soft verifiers are noisy; results are on a single model family (Qwen3); benchmark contamination risk from automated hard-problem generation; and a heavy data funnel (400K → 16.5K).

Conclusion

VHG's core lesson generalizes beyond math problem generation: any proxy reward targeting "difficulty" must be gated by the more fundamental constraint of "correctness."

Paper: https://arxiv.org/abs/2605.06660