[论文] TTPO: Test-Time Policy Optimization

论文概要

研究领域: NLP 作者: Aozhe Wang, Zhengxi Lu, Jianze Wang 发布时间: 2026-08-28 arXiv: 2508.11369

中文摘要

近期突出的训练后方法，如强化学习（RL）和策略内自蒸馏（OPSD），推动了大语言模型数学推理能力的快速进步，但它们对真实标签的依赖排除了测试时训练（TTT）的可能性。用多数投票伪标签替代真实标签是一种自然选择，但它很脆弱：一次错误的投票会腐化教师模型并误导每个token。我们观察到这种失败模式是不对称的：与伪标签不一致的rollout通常是错误的，无论投票本身是否正确。基于这一观察，我们提出测试时策略优化（TTPO），一种不对称目标——通过OPSD蒸馏一致的rollout，并用分组RL惩罚不一致的rollout。Token级选择进一步细化两个分支：蒸馏对已收敛位置降权，而RL仅惩罚置信错误。即使在频繁的伪标签错误下，两种更新都保持良定义，且多数投票路由在模型改进时产生更紧的自监督。无需任何标签，TTPO在五个竞赛级基准上与有标签监督的OPSD持平，在TTT中将Qwen3-1.7B从38.0%提升到45.2%，无思考模式下提升+25.2%到+36.4%，并展现出强大的跨任务泛化能力。

原文摘要

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. ...

#论文 #arXiv #NLP #小凯