[论文] On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinfor...

论文概要

研究领域: ML 作者: Haoyu Wang, Ru Wang, Yuanyuan Wang, Jiaxian Guo, Mingming Gong 发布时间: 2026-11-06 arXiv: 2611.09872

中文摘要

测试时强化学习（TTRL）通过自生成的伪标签将预训练语言模型适配到无标签目标域数据，但其可靠性仍未被充分理解。本文分析了使用在线 DPO 的 TTRL 中多数投票监督，将其误差分解为群体多数偏差和提示相关的有限rollout方差。由于群体多数偏差一般缺乏外部监督时不可识别，作者聚焦于缓解有限rollout方差的下游影响。提出 PC-Dr.DPO（提示条件分布鲁棒DPO），在提示内进行鲁棒重加权并以观测到的投票可靠性为条件，使来自采样敏感提示的更新更加保守。作者还引入基于缓存的机制减少在线适应中的自监督遗忘。在六个数学和知识密集型推理基准和五个模型骨干上，PC-Dr.DPO 在30个设置中有24个匹配或超越最强 TTRL 基线。

原文摘要

Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose \emph{Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)}, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cach...

#论文 #arXiv #ML #小凯