ANCORA: Teaching AI to Write Its Own Exams — From Answering to Question-Asking

Forum topic · 小凯 · 2026-05-01

Summary

A zhichai.net forum post analyzes ANCORA (Anchored-Curriculum framework), a reinforcement learning framework from Wuhan University (arXiv:2604.27644) that turns a language model from an answerer into a question proposer. One shared policy plays two roles: a Proposer generates new formally verifiable specifications, and a Solver attempts them with deterministic compiler 0/1 verification. Three pillars enable this: two-level group-relative updates coupling Proposer and Solver advantages with MLRL weighting that emphasizes rare successes; a 'manifold projection' stage using iterative self-distillation SFT to anchor the model on the valid output manifold (raising Dafny2Verus pass@1 from 0.4% to 26.6% before RL, preventing manifold collapse); and a UCB-selected curriculum DAG of validated new questions with MinHash deduplication and solved-only admission, provably expanding beyond the initial seed support. The Proposer's Band-1-of-K reward maximizes value when the Solver succeeds exactly once in eight attempts, targeting questions at the model's capability frontier. On Qwen2.5-Coder-3B with ~200 GPU-hours on two A100s, ANCORA reaches 81.5% pass@1 on Dafny2Verus (0-shot), 44.1% on MBPP, and transfers to HumanEval at 17.2% with no additional training. The post frames ANCORA as a path for continued AI improvement beyond human-labeled data exhaustion, while noting risks of noisy verifiers in less formal domains.

ANCORA: Teaching AI to Write Its Own Exams

*Author: Xiaokai | Source: arXiv:2604.27644v1 [cs.LG] | Institution: Wuhan University*

Key points

ANCORA (Anchored-Curriculum framework) lets the same model act as both Proposer (generating new, verifiable specifications/questions) and Solver (attempting them, with compiler-based 0/1 hard verification), trained jointly within one RL loop sharing a single policy πθ.
The author argues existing paradigms hit a ceiling: SFT caps at human-level imitation with data running out, and RLVR (used by DeepSeek-R1, OpenAI o1) only polishes solving skills on a fixed question bank — no new questions, no new capabilities.

Pillar 1: Two-level group-relative updates

Extends GRPO: for each seed question, the Proposer generates N candidate questions; each is judged by the Solver's K attempts, and advantages from both levels are coupled into one update.
Uses MLRL (Maximum Likelihood RL) alignment instead of flat REINFORCE weighting: rare successes get proportionally higher gradient weight (e.g., 1 success in 8 → weight 8), steering learning toward the "difficult but reachable" region.

Pillar 2: Manifold projection — anchor before walking

In formal verification, valid outputs are extremely sparse; naive RL causes manifold collapse, as sparse 0/1 rewards push the model off the valid manifold.
Fix: iterative self-distillation SFT first — seed with human-written specs, then distill back self-generated, validator-passing spec+implementation pairs. This raised Dafny2Verus pass@1 from 0.4% to 26.6% in three iterations.
Per the paper's failure analysis (Appendix B), without this projection the Proposer collapses regardless of entropy reward choice (Bernoulli variance, exponential decay, Band-1-of-K).

Pillar 3: UCB curriculum DAG

Validated new questions are re-inserted into a dynamic seed pool, forming a DAG rooted in human seeds, filtered by: syntax checks, MinHash novelty (near-duplicates with Jaccard ≥ 0.70 removed), and solved-only admission (a question must be solved at least once to enter the pool).
Seed selection uses MCTS-style UCB, preferring high-success, under-explored nodes — a curriculum in the Zone of Proximal Development.
Proposition 4.1: if each iteration has at least one seed that can generate a valid new question with positive probability, the DAG grows almost surely unbounded — training distribution strictly exceeds the seed support.

Band-1-of-K reward

The Proposer is rewarded maximally when the Solver succeeds exactly once in K=8 attempts — indicating the question sits precisely at the Solver's capability boundary (hard but valid), i.e., maximal information gain.

Experiments (Qwen2.5-Coder-3B, 2×A100, ~200 GPU-hours, Verus setting)

| Method | Dafny2Verus pass@1 | MBPP pass@1 | HumanEval pass@1 | |--------|--------------------|-------------|------------------| | AlphaVerus (50-shot, no training) | 30.7% | 20.2% | 14.1% | | SFT baseline | 26.6% | 8.8% | 6.2% | | RFT | 41.2% | 28.3% | 12.0% | | PSV (1-shot) | 65.7% | 36.8% | 19.1% | | ANCORA (0-shot) | 81.5% | 44.1% | 19.2% |

0-shot ANCORA beats 1-shot PSV by 15.8 points on Dafny2Verus — capability is internalized, not format mimicry.
SFT→ANCORA is ~3× improvement; the 26.6% manifold-projection starting point is a prerequisite, not a bonus.
MBPP pass@1→pass@10 rises only 44.1%→47.9%: sharpening, not expansion — near-deterministic solving of known-solvable problems.
Transfer (trained only on Dafny2Verus, zero extra training): MBPP 36.2% (vs PSV 25.3%), HumanEval 17.2% (vs PSV 6.8%) — indicating transferable question-generation/solving structure.

Why it matters

With human-generated text data nearing exhaustion (Villalobos et al., 2024), ANCORA offers a route to autonomous knowledge expansion: AI generates its own curriculum, verified by deterministic compilers, provably beyond the seed distribution.
Caveats: the self-play loop depends on a clean verifier (Verus is deterministic and perfect — rare in the real world); noisy verifiers could amplify bias or circularity. Whether the framework extends to fuzzier domains (e.g., creative writing) remains open.

One-line takeaway

Teaching AI to write exams may matter more than teaching it to answer them: a model that precisely poses "just solvable" challenges no longer needs humans to supply new question banks.

References

Paper: https://arxiv.org/abs/2604.27644
Code: GitHub link mentioned in the paper but not fully provided.