From Answer Machine to Question Setter: How ANCORA Teaches AI to Examine Itself
> Author: Xiaokai | Source: arXiv:2604.27644v1 [cs.LG] | Institution: Wuhan University
A Counterintuitive Question
If a student solves a thousand problems with 95% accuracy, have they truly learned the subject? Real mastery shows when they can create ten good problems themselves—problems that expose conceptual blind spots and impress teachers. Writing good problems is far harder than solving them, because it requires standing at a higher vantage point over the knowledge structure.
ANCORA's core ambition: evolve language models from "answerers" into "question setters."
The Hidden Ceiling of Current RL
Current LLM training is essentially "drill + check answers":
1. SFT (Supervised Fine-Tuning): imitating human answers—the ceiling is human level, and human data is running out. 2. RLVR (RL from Verifiable Rewards): fixed question banks, binary rewards, the path of DeepSeek-R1 and OpenAI o1.
RLVR's hidden flaw: the question bank is static. The model polishes solutions to known problems but never creates new, valuable challenges. Without new problems, there are no new capabilities—the model's ceiling is locked by the initial bank.
ANCORA's Three Pillars
ANCORA lets one policy πθ play two roles:
- Proposer: generates new, verifiable specifications from existing problems
- Solver: attempts them; a compiler gives hard 0/1 verdicts
- Transfer MBPP pass@1: 36.2% (PSV: 25.3%)
- Transfer HumanEval pass@1: 17.2% (PSV: 6.8%; SFT: 6.2%)
- Paper: https://arxiv.org/abs/2604.27644
- Code: GitHub (mentioned in the paper; full link not provided)
Pillar 1: Two-Level Group-Relative Updates
Traditional GRPO samples N answers for one fixed problem. ANCORA extends this to two levels: for each seed problem, the Proposer generates N candidate problems, each judged by the Solver's K attempts. Advantage signals from both levels are coupled into the same policy update. A problem's quality is decided by the Solver's real performance—not by humans.
A key technique here is MLRL (Maximum Likelihood RL): instead of REINFORCE's flat 1/N gradients, success probability-weighted gradients give rare successes higher weight (e.g., one success in eight samples gets weight 8). The model thus learns from "difficult but reachable" regions rather than drilling the comfort zone.
Pillar 2: Manifold Projection — Stand Firm Before Walking
In formal verification, valid outputs are extremely sparse. Naive RL on sparse rewards causes Manifold Collapse: occasional lucky successes push the model toward directions surrounded by cliffs of syntax errors.
ANCORA's pragmatic fix: anchor the model onto the valid manifold first via iterative self-distillation SFT. Seed specifications teach basic format (0.4% → ~0.5%), then early ANCORA iterations collect model-generated specification+implementation pairs that pass verifiers, deduplicate, and distill them back into SFT. After three rounds, Dafny2Verus pass@1 rises from 0.4% to 26.6%—enough coverage for stable RL. Appendix B shows that without this projection, the Proposer collapses regardless of entropy reward choice (Bernoulli variance, exponential decay, Band-1-of-K).
Pillar 3: UCB Curriculum DAG — A Self-Growing Question Bank
Verified new problems are reinserted into a dynamic seed pool, forming a directed acyclic graph rooted in human-written seeds. New nodes must pass three filters:
1. Format filtering (syntax check) 2. MinHash novelty check (near-duplicates with Jaccard ≥ 0.70 removed) 3. Solver validation ("solved-only admission": at least one Solver success)
Seed selection uses MCTS-style UCB: prioritizing high-success, under-explored nodes—i.e., the Zone of Proximal Development. Proposition 4.1 proves that if at least one seed can generate a valid new problem with positive probability each iteration, the curriculum DAG grows unboundedly almost surely. The training distribution strictly exceeds the initial seeds' support.
Band-1-of-K: Rewarding "Just Solvable"
The Proposer is rewarded neither for problems the Solver always solves (too easy) nor never solves (no signal). ANCORA's main run uses Band-1-of-K: full reward only if the Solver succeeds exactly once in K=8 attempts.
Why exactly once? MLRL gradients diverge most at K=1—rare successes get the highest weight. This corresponds to problems sitting at the Solver's capability boundary: hard enough to be missed 7 times out of 8, yet that one success proves the problem is valid. Such problems carry maximum information gain.
Experiments: A 3B Model Punches Up
ANCORA was validated on the Verus formal verification environment with Qwen2.5-Coder-3B (only 3B parameters), 2× A100, ~200 GPU hours.
Test-Time Training (in-distribution)
| Method | Dafny2Verus pass@1 | MBPP pass@1 | HumanEval pass@1 | |--------|--------------------|-------------|------------------| | AlphaVerus (50-shot, no training) | 30.7% | 20.2% | 14.1% | | SFT baseline | 26.6% | 8.8% | 6.2% | | RFT (rejection fine-tuning) | 41.2% | 28.3% | 12.0% | | PSV (1-shot) | 65.7% | 36.8% | 19.1% | | ANCORA (0-shot) | 81.5% | 44.1% | 19.2% |
Highlights:
1. 81.5% vs 65.7%: ANCORA at 0-shot beats 1-shot PSV by 15.8 points—the model internalizes genuine skill rather than example-format mimicry. 2. 26.6% → 81.5%: nearly 3× over the SFT baseline; that 26.6% is the post-projection starting line. RL from 0.4% would have collapsed. 3. MBPP 44.1% → 47.9% (pass@1→pass@10): a very flat gain, indicating near-deterministic solutions to solvable problems—sharpening, not expansion.
Transfer (cross-distribution)
Trained only on Dafny2Verus, tested directly:
ANCORA learns transferable problem-generation and solving structure, not Dafny2Verus templates.
Why This Matters
ANCORA addresses a fundamental question: when human-labeled data runs out, how does AI keep improving? If AI can only learn what humans wrote, its ceiling is the sum of human knowledge. ANCORA offers a way out: AI can set its own questions, verify them, and learn—not data augmentation, but autonomous knowledge expansion, guaranteed by Proposition 4.1.
Risks remain: Verus's deterministic 0/1 verdicts are unusually clean feedback. With flawed verifiers, self-play could amplify bias or loop meaninglessly. ANCORA's filters (solved-only admission, MinHash dedup, UCB exploration) constrain this, but extension to fuzzier domains like open-ended creative writing is an open question.
One-Sentence Takeaway
Teaching AI to write problems is more promising than teaching it to solve them. When a model can stand at the frontier of the knowledge manifold and pose challenges that are "just barely solvable," it no longer needs humans to supply new question banks—it grades its own exams and climbs out of its own traps. Isn't that exactly what all good teachers do?