An exam without an answer key — RL has always rested on one step: grade against the correct answer. On August 27, 2026, ZJU-REAL (Zhejiang University) and Alibaba removed that step. Their paper, TTPO: Test-Time Policy Optimization (arXiv:2608.27448), lets a model take AIME-style problems, grade each other's work by majority vote, and update itself in place — with no one ever supplying the right answer. Strikingly, label-free TTPO matches the label-trained teacher, and in one setting even beats a version trained with real labels.
1. An exam that ships no answer key
Test-time training (TTT) uses the exam itself as the training set — but exam papers never come with labels. The two existing signal sources both eat labels: GRPO, the strict referee that scores against ground truth, and OPSD (online policy self-distillation), the tutor that has the same model re-solve the problem with the answer in its prompt and thinking mode on, teaching the non-thinking self token by token. Prior label-free work, TTRL, used majority voting over sampled answers as pseudo-labels — but on competition-level problems, the majority vote is often wrong.
2. An 85%-wrong vote, and one reliable negative signal
On AIME 2026, Qwen3-1.7B with 64 samples produces a wrong majority vote on ~85% of problems. That looks like a death sentence — until you look under the wrong votes: among problems where the pseudo-label is wrong, responses that *disagree* with the vote are themselves wrong 79% of the time.
Judging "inconsistent" requires no answer content — the verdict holds whether the vote is right or wrong. Punishing disagreement is far safer than rewarding agreement. This echoes negative learning from noisy-label image classification (ICCV 2019), imported into LLM test-time training. The other danger lies in the distillation pipeline: a wrong vote as reward misleads once per response; a wrong vote as a dense distillation teacher misleads every token. Prior U-OPSD distilled precisely the inconsistent responses — putting the most dangerous signal into the most expensive pipeline.
3. Division of labor: each signal where it is reliable
TTPO samples 64 responses per problem, clusters them by mathematical equivalence, and takes the largest cluster's answer as the pseudo-label. Then:
- Consistent (positive) responses go to the distillation school. The teacher is the same model with the pseudo-label in its prompt and thinking mode on; the student has thinking off, differing only by a prefix. Distillation is token-weighted: high weight where the student hesitates or confidently diverges from the teacher, near-zero where both have converged. Even if the vote is wrong, the teacher can only teach *how thinking mode approaches the problem* — worst case harmless, best case standard OPSD. Bad votes are structurally contained on this branch.
- Inconsistent (negative) responses go to the punishment school. GRPO penalizes disagreement with the vote, plus a safeguard: only the top-half "confidently wrong" tokens (low-probability but high-certainty) receive updates. This safeguard is essential — in the asymmetric setup, no positive advantage gradient offsets a wrongly punished locally-correct reasoning step.
- Reporting. Both TTPO and baselines report peak checkpoints (every 25 steps) — peak-vs-peak flatters both sides.
- Distillation accounting. With 85% wrong votes, most positive-branch distillation is "thinking-to-non-thinking format transfer." How much of the +36.4 non-thinking gain is format absorption vs. new reasoning is not decomposed.
- Compute. 64 samples × up to 16K tokens per problem — TTT hides a large inference bill, with no time/compute comparison given.
- Scope. All evidence is from short-answer math competitions; no code or open-ended reasoning. Training code is released, but no checkpoints and no license file.
- Paper: Wang, A., Lu, Z., et al. (2026-08-27). TTPO: Test-Time Policy Optimization. arXiv:2608.27448.
- Project page · Code (ZJU-REAL/TTPO) · HF Papers discussion
- Related: TTRL (NeurIPS 2026) · OPSD (arXiv:2601.18734) · U-OPSD (arXiv:2608.06296) · Negative Learning (ICCV 2019)
The two branches are joined with a coefficient of 0.1 (raw GRPO gradients are an order of magnitude larger than the distillation loss). Training runs on 4×H20 GPUs with LoRA, a single node — the baselines used 8. An ablation warning worth copying out: reversing the assignment (GRPO punishes agreement, distill disagreement) crashes the score from 48.9 to 37.2.
4. Wrong votes beating true labels, and a self-raising ceiling
Label-free test-time training, averaged over five benchmarks (AIME25/26, HMMT25/26, BRUMO25), Avg@12:
| Model | Start | TTRL | TTPO | Gain | |---|---|---|---|---| | Qwen3-1.7B | 38.0 | 40.2 | 45.2 | +7.2 | | Qwen3-4B | 57.4 | 58.8 | 61.1 | +3.7 | | Qwen3-8B | 60.7 | 63.0 | 65.3 | +4.6 |
(Trained 4B at 61.1 overtakes the untrained 8B at 60.7.) With labeled data, real-label OPSD moves 1.7B from 34.6 to 39.7; label-free TTPO reaches 40.1 — a match with a slight edge. The biggest gain is in non-thinking mode: 8B jumps from 20.3 to 56.7 (+36.4), because punishment lands directly on the student's own generation distribution.
Two counterintuitive findings stand out:
1. Pseudo-labels beat real labels. Swapping the majority vote for true answers *lowers* performance. On hard problems almost no response matches the truth — the distillation branch starves and the GRPO group advantage computes to zero. The majority vote always yields a consensus, keeping both branches alive and exploration entropy high. 2. The ceiling raises itself. The training signal comes from majority votes, so the initial ceiling is the model's current majority-vote level. As training pushes past it, responses get stronger, votes get more accurate, and the ceiling rises. Single-benchmark training transfers to other benchmarks — the gains are reasoning, not memorization.
5. Cold water: four places to read with skepticism
6. The landscape
By 2026 this track is crowded: TTRL (NeurIPS 2026) proved majority votes can serve as rewards; OPSD (arXiv:2601.18734) showed self-distillation teaches finer than scalar rewards; then three "unsupervised OPSD" variants raced concurrently — Gkountouras et al. distilled all responses with consensus as privileged context, U-OPSD (arXiv:2608.06296) distilled only inconsistent responses, and TTPO offers a third split: consistent → distill, inconsistent → punish. Corresponding authors: Shen Yongliang (ZJU) and Chen Qianglong (Alibaba); co-first authors Wang Aozhe and Lu Zhengxi.
TTPO's answer in one sentence: when the correct answer is absent, "being against the consensus" remains a reliable indictment — punishing it is safer than rewarding anything. Learning doesn't always need an omniscient teacher; sometimes it just needs a room of similarly-skilled classmates, a rule that expels outliers, and a teacher forbidden from premature certainty.
The vote can be 80% wrong; the punishment is still nearly always right — failure was never symmetric.