From Problem Solver to Problem Setter: How ANCORA Teaches AI to "Test Itself"
> Author: Xiaokai | Source: arXiv:2604.27644v1 [cs.LG] | Institution: Wuhan University
1. A Counterintuitive Question
Imagine teaching a child math. You give them a thousand problems; they finish them all with 95% accuracy. Have they really learned?
Most likely they've learned "problem-solving patterns," not mathematics itself. The real test of understanding isn't solving another thousand similar problems—it's whether they can write ten good problems of their own: ones that stump classmates, expose conceptual blind spots, and win a teacher's nod.
Writing a good problem is ten times harder than solving one. You must stand at a higher vantage point on the knowledge structure and see where the traps and fault lines are.
That is ANCORA's core ambition: evolving language models from "answerers" into "question-writers."
2. The Hidden Ceiling of Existing RL
Today's LLM training paradigms boil down to "drill problems + check answers":
1. SFT (Supervised Fine-Tuning): imitate human-written gold answers. The ceiling is human-level, and human data is running out. 2. RLVR (RL from Verifiable Rewards): a fixed problem bank, the model tries repeatedly, correct answers earn rewards. DeepSeek-R1 and OpenAI o1 both take this route.
RLVR's problem is subtle: the problem bank is static. The model merely polishes "solution techniques for known problems"—it never creates new, valuable hard problems. Like a student who has drilled ten years of past exam papers to perfection but never wonders "if I set the exam, how would I lay traps?"
The deeper issue: no new problems, no new capabilities. The model's capability boundary is locked by the initial problem bank's ceiling.
ANCORA aims to break that lock.
3. ANCORA's Three Pillars
ANCORA (Anchored-Curriculum framework) lets a single model play two roles:
- Proposer: given existing problems, generate new verifiable specifications/problems
- Solver: attempt the Proposer's problems, with a compiler giving hard 0/1 judgments
- Proposer level: for each seed problem, generate N candidate new problems; each one's quality is determined by the Solver's K attempts on it.
- Solver level: for each new problem, the Solver attempts K solutions.
- Transfer MBPP pass@1: 36.2% (vs PSV's 25.3%)
- Transfer HumanEval pass@1: 17.2% (vs PSV's 6.8%)
- Paper: https://arxiv.org/abs/2604.27644
- Code: GitHub repository mentioned in the paper (full link not provided)
Both roles share one policy πθ and are trained simultaneously through the same RL loop. The Proposer's quality isn't judged by humans but by the Solver's real-world results. Problems the Solver barely solves may earn the Proposer the highest score—because "just barely solvable" problems carry the most teaching value.
3.1 Pillar One: Two-Level Group-Relative Updates
Standard GRPO samples N answers for a fixed problem and computes relative advantages. ANCORA extends this to two levels:
Advantage signals from both levels are coupled to update the same policy. A key trick here is MLRL (Maximum Likelihood RL) alignment: whereas REINFORCE gives each sample a flat 1/N gradient, MLRL weights by success rate—rare successes get higher gradient weight (e.g., 1 success in a group of 8 gets weight 8). This makes the model genuinely learn toward the "hard but reachable" region instead of over-drilling its comfort zone.
3.2 Pillar Two: Manifold Projection—Stand Firm Before Walking
In formal verification, valid outputs are extremely sparse. The Proposer must generate specifications that pass the compiler's syntax checks; the Solver must write implementations that pass verification. Valid specification+implementation pairs are a drop in the ocean of the generation space.
Running RL directly on such sparse rewards causes Manifold Collapse: the model wanders off the valid region, pushed by noisy 0/1 gradients with no signal to keep it stable.
ANCORA's pragmatic fix: anchor the model onto the valid manifold first via iterative self-distillation SFT.
1. SFT on a small set of human-written seed specifications (0.4% → ~0.5% pass@1). 2. Run early ANCORA iterations, collect model-generated specification+implementation pairs that pass the verifier, deduplicate, and distill them back into SFT data. 3. After three rounds, Dafny2Verus pass@1 rises from 0.4% to 26.6%.
That 26.6% means the model is "standing on the valid manifold"—about a quarter of its generations pass verification—enough coverage for subsequent RL to train stably. This is not an optional trick but a necessary precondition: the paper's failure analysis (Appendix B) shows that without manifold projection, the Proposer collapses regardless of the entropy reward used (Bernoulli variance, exponential decay, Band-1-of-K).
3.3 Pillar Three: The UCB Curriculum DAG—A Strictly Curated Self-Growing Problem Bank
The Proposer can only make local mutations of seeds. To reach distant knowledge territory, ANCORA uses composition: verified new problems are re-inserted into a "dynamic seed pool" as future material. These seed nodes form a directed acyclic graph (DAG), rooted at human-written initial seeds, with children admitted after triple filtering:
1. Format filter: syntax check to reject obviously invalid specifications 2. MinHash novelty check: near-duplicates with Jaccard ≥ 0.70 are removed 3. Solver verification: only problems solved at least once by the Solver enter the bank ("solved-only admission")
Seed selection uses MCTS-style UCB (Upper Confidence Bound), prioritizing nodes with "high success rates but insufficient exploration"—like a good teacher picking problems in the Zone of Proximal Development rather than repeating easy ones or fixating on impossible ones.
The paper proves (Proposition 4.1): as long as at least one seed per iteration can generate a valid new problem with positive probability, the curriculum DAG's node count grows to infinity almost surely. In other words, ANCORA's training distribution strictly exceeds the support of the initial seeds—the model is creating problems beyond its own training data.
4. Band-1-of-K: Rewarding "Just Barely Solved"
The Proposer's reward is finely tuned. It does not reward problems the Solver always solves (too easy, no learning value) nor problems it always fails (invalid or uninformative).
ANCORA's main run uses the Band-1-of-K reward: the Proposer gets full marks if the Solver succeeds exactly once in K=8 attempts.
Why exactly once? Because MLRL's gradient peaks when success is rarest—a problem at the Solver's capability boundary is hard enough to be solved only 1-in-8 times, yet the single success proves the problem is valid. This essentially rewards problems with maximum information gain: an all-solve problem says "you already know this"; an all-fail problem says little (it may just be buggy); a "1 of 8" problem says "there is a narrow but real path here"—the most precious learning signal.
5. Experiments: A 3B-Parameter Underdog
ANCORA was validated on the Verus formal verification environment using Qwen2.5-Coder-3B (only 3B parameters), 2× A100 GPUs, roughly 200 GPU-hours of training.
Test-Time Training Setting (In-Distribution)
| Method | Dafny2Verus pass@1 | MBPP pass@1 | HumanEval pass@1 | |--------|--------------------|-------------|------------------| | AlphaVerus (50-shot, no training) | 30.7% | 20.2% | 14.1% | | SFT baseline | 26.6% | 8.8% | 6.2% | | RFT (Rejection Fine-Tuning) | 41.2% | 28.3% | 12.0% | | PSV (1-shot inference) | 65.7% | 36.8% | 19.1% | | ANCORA (0-shot inference) | 81.5% | 44.1% | 19.2% |
Numbers worth savoring:
1. 81.5% vs 65.7%: ANCORA at 0-shot beats PSV's 1-shot by 15.8 points—ANCORA learned internalized generate-and-solve capability, not just example-format mimicry. 2. 26.6% → 81.5%: nearly 3× over the SFT baseline. That 26.6% was the post-manifold-projection starting line; running RL from 0.4% would have collapsed. 3. MBPP 44.1% → 47.9% (pass@1 → pass@10): a very flat curve, indicating near-deterministic solutions to solvable problems—sharpening rather than expansion.
Transfer Setting (Cross-Distribution)
Trained only on Dafny2Verus, tested directly on MBPP and HumanEval (zero additional training):
This shows ANCORA learns a transferable problem-generation and solving structure, not a Dafny2Verus-specific template. Despite the stylistic gap between Python tasks and Dafny2Verus algorithmic verification (17.2% vs 36.2%), that's still nearly 3× the SFT baseline's 6.2%.
6. Why This Matters
ANCORA's value goes beyond "another formal verification SOTA."
It addresses a fundamental question: when human-labeled data runs out, how does AI keep improving?
Villalobos et al. (2024) predict that publicly available human-generated text may soon become a bottleneck. If AI can only learn what humans have written, its ceiling is the sum of human knowledge. ANCORA offers a way out: AI can write its own problems, verify them itself, and learn by itself.
This is not mere "data augmentation" (transforming existing data) but autonomous knowledge expansion. Proposition 4.1 guarantees the Proposer's problems strictly exceed the initial seeds' support, and the Solver's gradients on those new problems push the policy into regions the original data never covered.
There are risks. If the verifier were flawed (Verus's compiler is deterministic and gives clean 0/1 judgments—rarely true in the real world), self-play could amplify bias or degenerate into meaningless loops. ANCORA's strict filtering (solved-only admission + MinHash dedup + UCB exploration) is an engineering attempt to constrain this, but extending to fuzzier domains like open-ended creative writing remains an open question.
7. One-Sentence Takeaway
ANCORA shows that teaching AI to write problems is more promising than teaching it to answer them. When a model can stand at the frontier of the knowledge manifold and precisely pose "just barely solvable" challenges, it no longer needs humans to feed it new problem banks—it writes its own exams, grades itself, and climbs higher through its own traps.
Isn't that exactly what every good teacher does?
---
References