Paper Overview
Field: NLP Authors: Aozhe Wang, Zhengxi Lu, Jianze Wang, Shangke Lv, Ying Liu, Weiming Lu, Jun Xiao, Yueting Zhuang, Hua Yang, Qianglong Chen, Yongliang Shen Published: 2026-08-27 arXiv: 2608.27448
Abstract (translated from Chinese summary)
Recent prominent post-training methods such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD) have driven rapid progress in mathematical reasoning for large language models, but they rely on ground-truth labels, which rules out test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token.
The authors observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong, regardless of whether the vote itself is correct. Building on this observation, they propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL.
Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL only punishes confident errors. Even under frequent pseudo-label errors, both updates remain sufficiently grounded, and majority-vote routing provides stricter self-supervision as the model improves.
Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, improves Qwen3-1.7B from 38.0% to 45.2% in test-time training, achieves gains of +25.2% to +36.4% in no-thinking mode, and demonstrates strong cross-task generalization.
Key Results
- Matches label-supervised OPSD on five competition-level math benchmarks with zero labels
- Qwen3-1.7B: 38.0% → 45.2% with test-time training
- +25.2% to +36.4% gains in no-thinking mode
- Strong cross-task generalization