TTPO: Test-Time Policy Optimization for Label-Free LLM Math Reasoning

Forum topic · 小凯 · 2026-08-29

Summary

TTPO (Test-Time Policy Optimization) is a new post-training method that enables test-time training for large language models without ground-truth labels. The paper, arXiv:2508.11369 by Aozhe Wang, Zhengxi Lu, and Jianze Wang, observes that majority-vote pseudo-labels used in test-time training fail asymmetrically: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this insight, TTPO applies an asymmetric objective—On-Policy Self-Distillation (OPSD) for rollouts that agree with the pseudo-label, and Grouped RL penalties for those that disagree—refined by token-level selection that downweights converged positions in the distillation branch and penalizes only confident errors in the RL branch. Without any labeled data, TTPO matches label-supervised OPSD on five competition-level math benchmarks, improves Qwen3-1.7B from 38.0% to 45.2% in test-time training, gains +25.2% to +36.4% in non-thinking mode, and shows strong cross-task generalization.

Overview

Field: NLP / LLM reasoning Authors: Aozhe Wang, Zhengxi Lu, Jianze Wang Published: 2026-08-28 arXiv: 2508.11369

Summary (translated from Chinese)

The authors observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, they propose Test-Time Policy Optimization (TTPO), an asymmetric objective that:

Distills agreeing rollouts via OPSD (On-Policy Self-Distillation)
Penalizes disagreeing rollouts with Grouped RL
Applies token-level selection to refine both branches: distillation downweights already-converged positions, while RL only punishes confident errors

Even under frequent pseudo-label errors, both updates remain well-defined, and majority-vote routing yields tighter self-supervision as the model improves.

Results

Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks
Improves Qwen3-1.7B from 38.0% to 45.2% in test-time training
Gains of +25.2% to +36.4% in non-thinking mode
Strong cross-task generalization

Full Abstract

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL.

*Auto-collected on 2026-08-29.*