Summary
Reinforcement learning with verifiable rewards (RLVR) has become the dominant paradigm for enhancing LLM reasoning, evolving from PPO to GRPO. This paper identifies limitations of GRPO's use of negative rollouts: negative samples lack gradients reflecting failure severity, and the vast combinatorial space makes penalizing sparsely sampled negative rollouts ineffective under binary rewards. The authors propose Positive-Only Policy Optimization (POPO), a novel RLVR framework where learning occurs exclusively from online positive rollouts via bounded importance sampling, with implicit negative gradients naturally emerging through reinforcement of positive probabilities via rollout reallocation. POPO stabilizes optimization using twin policy networks with momentum-based adaptive update rules, and replaces KL divergence with a bounded similarity penalty in twin representation space. Experiments with open-source Qwen-series models on math benchmarks show POPO matches or exceeds GRPO, notably achieving 36.67% on AIME 2025 with Qwen-Math-7B versus GRPO's 30.00%. Ablation and sweep studies confirm the necessity and robustness of POPO's components. Paper: arXiv 2605.06650.
Paper Overview
Field: NLP
Authors: Mingwei Xu, Hao Fang
Published: 2026-05-07
arXiv:
2605.06650Abstract
Reinforcement learning with verifiable rewards (RLVR) has become the dominant paradigm for enhancing the reasoning capabilities of large language models (LLMs), due to the determinism of verification. The community has witnessed a rapid shift from Proximal Policy Optimization (PPO) to Group Relative Policy Optimization (GRPO), which replaces complex advantage estimation with a simple estimate over grouped positive and negative rollouts. However, the authors observe that negative rollouts may lack gradients reflecting failure severity, and the sheer size of the combinatorial space makes penalizing sparsely sampled negative rollouts unlikely to cover meaningful reward signal under sparse binary rewards.
In this work, the authors propose Positive-Only Policy Optimization (POPO), a novel RLVR framework in which learning can occur solely from online positive rollouts. Specifically, POPO uses bounded importance sampling over the positive rollout set, so no disjoint negative rollouts are used for gradient guidance. They show that by reinforcing positive probabilities through rollout reallocation, implicit negative gradients emerge naturally.
POPO then stabilizes policy optimization through two mechanisms:
- It applies twin policy networks and a momentum-based adaptive update rule for stable policy evolution.
- It replaces KL divergence with a bounded similarity penalty term in the twin representation space.
Extensive experiments on math benchmarks at various levels, using publicly available and mature text LLMs such as the Qwen series, demonstrate that POPO achieves performance comparable to or better than GRPO. Notably, POPO reaches
36.67% on AIME 2025 with Qwen-Math-7B, outperforming GRPO's 30.00%. Ablation and sweep studies further illustrate the necessity and robustness of POPO's components.
---
*Auto-collected on 2026-05-10*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177619696