[论文] TTPO: Test-Time Policy Optimization

论文概要

研究领域: NLP 作者: Aozhe Wang, Zhengxi Lu, Jianze Wang, Shangke Lv, Ying Liu, Weiming Lu, Jun Xiao, Yueting Zhuang, Hua Yang, Qianglong Chen, Yongliang Shen 发布时间: 2026-08-27 arXiv: 2608.27448

中文摘要

近期突出的后训练方法如强化学习（RL）和在线策略自蒸馏（OPSD）推动了大语言模型数学推理的快速进步，但它们依赖真实标签，排除了测试时间训练（TTT）的可能性。用多数投票伪标签替代真实标签是一种自然选择，但它很脆弱：错误的投票会污染教师模型并误导每个token。我们观察到这种失败模式是不对称的：与伪标签不一致的rollout通常是错误的，无论投票本身是否正确。基于这一观察，我们提出了测试时间策略优化（TTPO），一种非对称目标：通过OPSD蒸馏一致的rollout，用分组RL惩罚不一致的rollout。Token级选择进一步细化两个分支：蒸馏降低已收敛位置的权重，而RL仅惩罚置信错误。即使在频繁的伪标签错误下，两种更新都保持充分的基础，且多数投票路由随着模型改进提供更严格的自监督。无需任何标签，TTPO在五个竞赛级基准上匹配标签监督的OPSD，在TTT中将Qwen3-1.7B从38.0%提升到45.2%，在无思考模式下获得+25.2%到+36.4%的提升，并显示出强大的跨任务泛化能力。

原文摘要

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. ...

#论文 #arXiv #NLP #小凯