ANCORA: Teaching LLMs to Generate Their Own Exam Questions via Self-Play RL
> Author: Xiaokai | Source: arXiv:2604.27644v1 [cs.LG] | Institution: Wuhan University
*(Full English translation of the original Chinese forum post.)*
1. A Counterintuitive Question
Imagine teaching a child math. You give them a thousand problems, they finish all of them with 95% accuracy. Do you think they've learned?
Most likely they've learned "problem-solving templates," not "math itself." The real test of understanding isn't doing another thousand similar problems — it's being able to write ten good problems yourself: ones that can stump classmates, expose conceptual blind spots, and win a teacher's nod.
Writing good problems is ten times harder than answering them well. You must stand above the knowledge structure and see where the traps and the watersheds are.
That is ANCORA's core ambition: evolve language models from "answerers" into "question writers."
2. The Invisible Ceiling of Existing RL
Current LLM training paradigms boil down to "grinding problems + checking answers":
1. SFT (Supervised Fine-Tuning): imitate human-written reference answers. The ceiling is human level, and human data is running out. 2. RLVR (RL from Verifiable Rewards): a fixed problem bank; the model tries repeatedly and gets rewards for correct answers. DeepSeek-R1 and OpenAI o1 both follow this path.
RLVR's problem is hidden: the problem bank is static. The model only polishes "solving skills on known problems" — it never creates new, valuable hard problems. It's a student who has memorized ten years of past exams but never thought about how they'd design traps if they were the examiner.
The deeper issue: without new problems, there are no new capabilities. The model's capability boundary is locked by the initial problem bank's ceiling.
ANCORA wants to break that lock.
3. ANCORA's Three Pillars
ANCORA (Anchored-Curriculum framework) has one model play two roles:
- Proposer: generates new, verifiable specifications/problems based on existing problems
- Solver: attempts the Proposer's problems, with a compiler giving a hard 0/1 verdict
- Proposer level: for each seed problem, generate N candidate new problems. Each new problem's quality is determined by the Solver's K attempts on it.
- Solver level: for each new problem, the Solver attempts K solutions.
- Transfer MBPP pass@1: 36.2% (vs PSV's 25.3%)
- Transfer HumanEval pass@1: 17.2% (vs PSV's 6.8%)
- Paper: https://arxiv.org/abs/2604.27644
- Code: https://github.com/ (a GitHub link is mentioned in the paper but no complete URL is given)
Both roles share one policy πθ and are trained simultaneously in one RL loop. There's inherent tension here — the model must learn "how to write problems" and "how to solve them," and its problems must be ones its current (or future) self can barely solve, but not too easily.
3.1 Pillar One: Two-Level Group-Relative Updates
Traditional GRPO (Group Relative Policy Optimization) samples N answers per fixed problem and computes relative advantages. ANCORA extends this to two levels:
The advantages at both levels are then coupled into a single update of the shared policy. Whether a Proposer's problem is good isn't decided by humans — it's decided by the Solver's real performance. Problems the Solver struggles with may earn the Proposer high scores, because "just barely solvable" problems have the most teaching value.
A key trick here: MLRL (Maximum Likelihood RL) alignment. Traditional REINFORCE gives every sample an equal gradient weight of 1/N; MLRL weights by success rate, so rare successes get higher gradient weight (e.g., if 1 of 8 in a group succeeds, that sample's weight is 8). This makes the model learn from the "hard but reachable" region rather than grinding fluency in its comfort zone.
3.2 Pillar Two: Manifold Projection — Stand Before You Walk
This is ANCORA's most striking diagnosis.
In formal verification, valid outputs are extremely sparse. The Proposer must generate specifications that pass a compiler's syntax checks; the Solver must write implementations that pass compiler verification. Valid specification+implementation pairs may be a drop in the ocean of the generation space.
Running RL directly on such sparse rewards causes Manifold Collapse.
Imagine a Proposer whose early problems are almost all syntactically invalid. It occasionally gets one right by luck, and the RL gradient pushes the model toward that direction. But that direction is surrounded by cliffs — the slightest deviation is a syntax error. Sparse 0/1 rewards can't provide enough gradient signal to "stand firm" on the valid manifold, so the model gets pushed off it and wanders in invalid space.
ANCORA's fix is pragmatic: first anchor the model onto the valid manifold via iterative self-distillation SFT.
Concretely: start with a small set of human-written seed specifications for SFT to teach basic format (0.4% → ~0.5%). Then run early ANCORA iterations, collect model-generated specification+implementation pairs that pass the verifier, deduplicate, and distill them back into SFT data. After three rounds, pass@1 on Dafny2Verus rises from 0.4% to 26.6%.
What does 26.6% mean? The model has "reached the valid manifold" — a quarter of its specifications pass verification. That coverage is enough for subsequent RL to run stably without being pushed off the cliff by sparse rewards.
This is not a nice-to-have trick — it's a necessary precondition. The failure analysis in Appendix B shows clearly: without manifold projection, the Proposer collapses regardless of which entropy reward is used (Bernoulli variance, exponential decay, Band-1-of-K).
3.3 Pillar Three: A UCB Curriculum DAG — A Strictly Vetted, Self-Growing Problem Bank
The Proposer can only make local mutations of a seed problem at a time. How do you explore distant knowledge regions?
ANCORA's answer is composition: verified new problems are re-inserted into a "dynamic seed pool" as material for future problem generation. These seed nodes form a directed acyclic graph (DAG) whose roots are human-written initial seeds and whose child nodes are model-generated, triple-filtered new problems:
1. Format filtering: syntax checks to remove clearly invalid specifications 2. MinHash novelty check: near-duplicates with Jaccard ≥ 0.70 are removed 3. Solver verification: only problems solved at least once by the Solver are admitted ("solved-only admission")
Seed selection uses an MCTS-style UCB (Upper Confidence Bound) algorithm: prefer nodes with "high success rate but insufficient exploration." Like a good teacher who neither reuses problems the students have mastered nor grinds on a problem the whole class fails, it selects in the Zone of Proximal Development.
The paper proves (Proposition 4.1): as long as at least one seed per iteration can generate a valid new problem with positive probability, the curriculum DAG's node count grows to infinity almost surely. This means ANCORA's training distribution strictly exceeds the support of the initial seeds — the model creates problems beyond its own training data.
4. Band-1-of-K: Rewarding "Just Barely Solved"
The Proposer's reward design is subtle. It does not reward problems the Solver solves entirely (too easy, no learning value), nor problems it fails entirely (invalid, no signal).
ANCORA's main run uses the Band-1-of-K reward: if the Solver succeeds exactly once in K=8 attempts, the Proposer gets full marks.
Why is "exactly once" optimal? Because the MLRL gradient formula diverges maximally at K=1 — rare successes get the highest weight. This corresponds exactly to a problem sitting at the Solver's capability boundary: hard enough that only 1 of 8 attempts succeeds, but not impossible, since that one success proves the problem is valid.
This is essentially rewarding maximum information gain. An all-correct problem says "you already know this" — no new information. An all-wrong problem says "dead end," but with limited information (possibly just a buggy problem). A "1 of 8" problem says "there is a narrow but genuinely traversable path here" — the most precious learning signal.
5. Experiments: A 3B-Parameter Comeback
ANCORA was systematically validated on the Verus formal verification setting, using Qwen2.5-Coder-3B (only 3 billion parameters), 2× A100, roughly 200 GPU-hours of training.
Test-Time Training setting (in-distribution)
| Method | Dafny2Verus pass@1 | MBPP pass@1 | HumanEval pass@1 | |--------|--------------------|-------------|------------------| | AlphaVerus (50-shot, no training) | 30.7% | 20.2% | 14.1% | | SFT baseline | 26.6% | 8.8% | 6.2% | | RFT (Rejection Fine-Tuning) | 41.2% | 28.3% | 12.0% | | PSV (1-shot inference) | 65.7% | 36.8% | 19.1% | | ANCORA (0-shot inference) | 81.5% | 44.1% | 19.2% |
A few numbers worth savoring:
1. 81.5% vs 65.7%: in a 0-shot setting (no examples given), ANCORA beats PSV's 1-shot (one example given) by 15.8 percentage points. This means ANCORA learned internalized question-writing + solving ability, not "mimicking example formats."
2. 26.6% → 81.5%: from SFT baseline to final ANCORA, nearly a 3× improvement. That 26.6% is the "starting line" after manifold projection — running RL directly from 0.4% would have collapsed the model.
3. MBPP 44.1% → 47.9% (pass@1→pass@10): an extremely shallow slope, indicating the model has formed "near-deterministic solutions" for solvable problems. This is sharpening rather than expansion — extreme confidence on known-solvable problems.
Transfer setting (out-of-distribution)
More striking is the transfer ability: trained only on Dafny2Verus, tested directly on MBPP and HumanEval (zero additional training):
This shows ANCORA learns transferable problem-generation and solving structure, not "Dafny2Verus solution templates." Although HumanEval's Python-style tasks differ considerably from Dafny2Verus's algorithmic verification (17.2% vs MBPP's 36.2%), it's still nearly a 3× improvement over the SFT baseline's 6.2%.
6. Why This Matters
ANCORA's value goes beyond "another formal verification SOTA."
It points at a more fundamental question: when human-annotated data runs out, how does AI keep evolving?
Villalobos et al. (2024) predicted that public human-generated text may soon become a bottleneck. If AI can only learn what humans have written, its ceiling is the sum of human knowledge. ANCORA offers a way out: AI can write its own problems, verify them itself, and learn from them.
This is not simple "data augmentation" (transforming existing data), but autonomous expansion of knowledge. The Proposer's problems strictly exceed the initial seeds' support (guaranteed by Proposition 4.1), and the gradient signals the Solver gains from these new problems drive the policy into regions never covered by the initial data.
There are risks, of course. If the verifier is flawed (the Verus compiler is deterministic and gives a perfect 0/1 verdict, but few real-world domains have such clean feedback), self-play may amplify biases or produce meaningless loops. ANCORA's strict filtering (solved-only admission + MinHash dedup + UCB exploration) is an engineering attempt to constrain this risk, but whether it extends to fuzzier domains (like open-ended creative writing) remains an open question.
7. One-Sentence Summary
ANCORA tells us: teaching AI to write exam questions is more promising than teaching it to answer them. When you can place a model at the frontier of the knowledge manifold and have it precisely throw out challenges that are "just barely solvable," it no longer needs humans to supply endless new problem banks — it writes its own exams, grades itself, and climbs higher out of its own traps.
Isn't that exactly what all good teachers do?
---
References