English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning (PC-Dr.DPO)

Forum topic · 小凯 · 2026-11-09

Summary

This post summarizes an arXiv paper (2611.09872) by Haoyu Wang, Ru Wang, Yuanyuan Wang, Jiaxian Guo, and Mingming Gong on the reliability of self-generated pseudo-labels in test-time reinforcement learning (TTRL), where pretrained language models are adapted to unlabeled target-domain data. The authors analyze majority-vote supervision under online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Since population-majority bias is generally unidentifiable without external supervision, they focus on mitigating finite-rollout variation by proposing Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which performs within-prompt robust reweighting conditioned on observed vote reliability, making updates from sampling-sensitive prompts more conservative. They also introduce a caching-based mechanism to reduce self-supervision forgetting during online adaptation. Across six math and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or surpasses the strongest TTRL baseline in 24 of 30 settings.

Paper Overview

Field: Machine Learning Authors: Haoyu Wang, Ru Wang, Yuanyuan Wang, Jiaxian Guo, Mingming Gong Published: 2026-11-06 arXiv: 2611.09872

Summary

Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, but the reliability of these labels remains poorly understood. This paper analyzes majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decomposes its error into two components:

  • Population-majority bias — generally unidentifiable without external supervision
  • Prompt-dependent finite-rollout variation — the focus of the paper's mitigation efforts
  • Method

    The authors propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which:

  • Performs within-prompt robust reweighting conditioned on observed vote reliability
  • Makes updates from sampling-sensitive prompts more conservative
  • Introduces a caching-based mechanism to reduce self-supervision forgetting during online adaptation

Results

Across six mathematics and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or surpasses the strongest TTRL baseline in 24 of 30 settings.

Original Abstract (excerpt)

> Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose *Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)*, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative...

---

*Auto-collected on 2026-11-10*

Tags

#test-time-reinforcement-learning#dpo#pseudo-labels#majority-voting#llm-reasoning#distributionally-robust-optimization#arxiv#machine-learning

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636981