TTPO: Test-Time Policy Optimization (arXiv:2608.27448) is a paper from Zhejiang University's ZJU-REAL lab and Alibaba, released 2026-08-27, tackling a seemingly impossible setting: test-time training (TTT) on competition problems without any ground-truth answers.
Key points
- The setting. Test-time training uses the exam questions themselves as the training set. Existing methods rely on labels: GRPO needs reference answers for rewards, and OPSD (online policy self-distillation) distills a thinking-mode teacher into a non-thinking student. TTRL instead uses majority voting over sampled rollouts as pseudo-labels — but on hard problems the votes are frequently wrong.
- The key statistic. On AIME 2026, Qwen3-1.7B with 64 rollouts gets the majority vote wrong on ~85% of questions. Yet among those mislabeled questions, 79% of rollouts that disagree with the wrong vote are themselves wrong. Judging "inconsistent with consensus" requires no answer key and is reliable whether the vote is right or wrong — failure is asymmetric, so supervision should be too. (This echoes negative learning from noisy-label image recognition, NLNL, ICCV 2019.)
- The method. Sample 64 rollouts per question, cluster by mathematical equivalence, take the largest cluster's answer as pseudo-label. Then:
- Consistent rollouts → distillation. The thinking-mode model (with pseudo-label in its prompt) teaches the non-thinking model token-by-token, weighted toward uncertain/disagreeing tokens. Even if the vote is wrong, distillation degenerates harmlessly into thinking-to-non-thinking transfer.
- Inconsistent rollouts → GRPO penalties, applied only to the top half of "confidently wrong" tokens (low-probability, high-certainty), since negative-sample penalties lack advantage-gradient counterbalancing.
- A 0.1 coefficient balances the two losses (raw GRPO gradients are ~10× larger). Training: 4×H20 GPUs, LoRA, single node. Reversing the assignment (punish consistent, distill inconsistent) collapses accuracy from 48.9 to 37.2.
- Results (average of five benchmarks: AIME25/26, HMMT25/26, BRUMO25, Avg@12):
- Two counterintuitive findings. 1. Pseudo-labels beat true labels. Swapping majority votes for real answers *lowers* performance: on hard problems few rollouts match ground truth, starving both the distillation branch and the penalty branch (zero within-group advantage). Votes always yield a consensus, keeping both branches active and exploration entropy high. 2. The ceiling rises. Once training saturates the current majority-vote ceiling, improved rollouts make votes more accurate, raising the ceiling again. Gains transfer across benchmarks, suggesting genuine reasoning improvement rather than memorization.
- Both TTPO and baselines report best-checkpoint (peak) scores — optimistic on both sides.
- Much of the +36.4 non-thinking gain may reflect format absorption from distillation rather than new reasoning ability; the accounting is not separated.
- High inference overhead (64 rollouts × 16k-token cap per question) is not benchmarked against baselines in time/compute.
- Evidence is limited to short-answer competition math; code and open-ended reasoning are untested. Released artifacts include training code but no checkpoints or license file.
- Paper: arXiv:2608.27448
- Project page: https://zju-real.github.io/TTPO/
- Code: https://github.com/ZJU-REAL/TTPO
- Related: TTRL (NeurIPS 2026), OPSD (arXiv:2601.18734), U-OPSD (arXiv:2608.06296), Negative Learning NLNL (ICCV 2019)
| Model | Base | TTRL | TTPO | Gain | |---|---|---|---|---| | Qwen3-1.7B | 38.0 | 40.2 | 45.2 | +7.2 | | Qwen3-4B | 57.4 | 58.8 | 61.1 | +3.7 | | Qwen3-8B | 60.7 | 63.0 | 65.3 | +4.6 |
Label-free TTPO (40.1) matches label-trained OPSD (39.7) on the 1.7B model. The largest gains come in non-thinking mode (8B: 20.3 → 56.7, +36.4).