R-Quest: Fixing Self-Questioning Loops That Degrade LLMs

The problem: self-evolution becomes self-degradation

In question-answer co-evolution frameworks, the best-known example is R-Zero: one model acts as a question generator, another as a solver, and they train each other with no human-labeled data. It sounds like a self-reinforcing positive loop. But when teams from Washington University in St. Louis and the University of Michigan ran the loop longer, they found it has an expiration date: by round five, fewer than half of the questions in the generation pool remain usable; after ten rounds, model performance drops 11.57 points below the no-training baseline. Self-evolution becomes self-degradation.

Their diagnosis and fix are described in the paper R-Quest (arXiv 2610.04299). Both root causes hide in question quality.

Root cause 1: bad questions accumulate

Manually checking 200 sampled questions per round: validity was 76.5% in round one, falling to 39.5% by round five — the questioner drifts, producing stems that lack conditions, are ambiguous, or are self-contradictory.

Surprisingly, R-Zero's built-in filter makes things worse. It keeps questions whose answer-consistency score falls between 0.3 and 0.8. Intuitively this should filter bad questions, but in tests, validity of retained questions dropped further to 27.64%. Why? Answer consistency only means a question has a well-defined answer — not that the question is well-formed. Several solver models can misread a broken question in the same way and agree perfectly. The filter removes "easy, good questions" and keeps "bad questions everyone gets wrong the same way."

Root cause 2: same wine, new bottle

Question generators repeat themselves: the same sequence-construction problem with a changed starting number or rephrased wording. R-Zero deduplicates with BLEU, an old machine-translation metric for surface similarity — it catches copy-paste, not disguised repeats. By round five, the top three question types made up 59.5% of the pool, and BLEU clustering shattered the largest type into 22 small clusters. Since dedup penalties scale with cluster size, fragmentation dilutes the penalty, letting reskinned duplicates re-enter training.

Are bad questions really the culprit?

A clean controlled experiment: same 1,273 questions, 766 good ones shared, 507 bad ones — used as-is on one side, repaired one-by-one (same knowledge point and difficulty) on the other. Average score across seven math benchmarks: 46.54 vs 49.91. Fixing only the bad questions, adding nothing new, gains 3.4 points. Bad questions are the disease, not background noise.

R-Quest's two gates

1. Quality gate: a one-time upfront cost — a strong model labels a batch of questions as a textbook, training the solver into a quality inspector that recognizes and rejects invalid questions. Every generated question passes through it; its verdict feeds the questioner's reward and filters solver training data. Note the nuance: "no longer relying on an external strong model" applies to loop operation — the upfront labeling cost is still paid once, not every round. 2. Novelty gate: a frozen base model compares new questions against the pool pairwise — judging task structure, not surface text. "Different numbers, same test" counts as a duplicate.

Only questions passing both gates enter training; rewards go only to valid, novel questions.

Results

• Fixed-round comparisons: across 12 benchmarks (seven math, two code, three general) and two model families (Qwen3-4B-Base and OctoThinker-3B), R-Quest leads on all, averaging 2-3 points above R-Zero.
• Ten-round long run: R-Quest climbs until round ten; R-Zero turns downward from round four. Final gap: 17.32 points — achieved while R-Zero had fallen 11.57 below its starting point.
• Counterintuitive ablation: freezing the inspector (no further evolution) costs 1.7 points — judgment itself must keep training inside the loop. R-Quest's question validity rises from 88% to 94-97.5%, versus R-Zero's fall from 76.5% to 39.5%.
• Reverse finding: gating novelty too strictly also hurts scores — math needs more repetitive practice than expected.

Availability

No code link is provided. For the sharpest figure, see Figure 1 on the arXiv page: two curves from the same starting point, one diving after five rounds, one still climbing at ten — the fork is at round four.

Paper: arXiv 2610.04299