Paper Overview
Field: Machine Learning Authors: Haoyu Wang, Ru Wang, Yuanyuan Wang, Jiaxian Guo, Mingming Gong Published: 2026-11-06 arXiv: 2611.09872
Summary
Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, but the reliability of these labels remains poorly understood. This paper analyzes majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decomposes its error into two components:
- Population-majority bias — generally unidentifiable without external supervision
- Prompt-dependent finite-rollout variation — the focus of the paper's mitigation efforts
- Performs within-prompt robust reweighting conditioned on observed vote reliability
- Makes updates from sampling-sensitive prompts more conservative
- Introduces a caching-based mechanism to reduce self-supervision forgetting during online adaptation
Method
The authors propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which:
Results
Across six mathematics and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or surpasses the strongest TTRL baseline in 24 of 30 settings.
Original Abstract (excerpt)
> Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose *Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)*, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative...
---
*Auto-collected on 2026-11-10*