POPO: Positive-Only Policy Optimization with Implicit Negative Gradients for RLVR

Forum topic · 小凯 · 2026-05-10

Summary

This forum post introduces POPO (Positive-Only Policy Optimization), a novel reinforcement learning with verifiable rewards (RLVR) framework for improving large language model (LLM) reasoning, proposed by Mingwei Xu and Hao Fang (arXiv: 2605.06650). The authors argue that in methods like GRPO, negative rollouts may carry no useful gradient signal, and the vast combinatorial space makes sparse-sampled negative rollouts ineffective under binary rewards. POPO enables learning exclusively from online positive rollouts using bounded importance sampling, with implicit negative gradients emerging naturally through rollout redistribution. Stability is achieved via a twin-policy network with momentum-based adaptive rules, and KL divergence is replaced by a bounded similarity penalty in twin representation space. Experiments on Qwen-series text LLMs across math benchmarks show POPO matches or exceeds GRPO, reaching 36.67% on AIME 2025 with Qwen-Math-7B versus GRPO's 30.00%. Ablation and sweep studies confirm the necessity and robustness of each component.

论文概要

Research Area: NLP Authors: Mingwei Xu, Hao Fang Published: 2026-05-07 arXiv: 2605.06650

Overview

Reinforcement learning with verifiable rewards (RLVR) has become the dominant paradigm for enhancing the reasoning capabilities of large language models (LLMs), thanks to the certainty of verification. The community has seen a rapid shift from Proximal Policy Optimization (PPO) to Group Relative Policy Optimization (GRPO), where GRPO replaces sophisticated advantage estimation with a simple estimate over grouped positive and negative rollouts.

However, the authors observe that negative rollouts may have no gradient reflecting the severity of failure, and the vastness of the combinatorial space makes penalizing sparsely sampled negative rollouts unlikely to cover meaningful reward signal under sparse binary rewards.

POPO: Positive-Only Policy Optimization

This work proposes POPO, a novel RLVR framework in which learning occurs exclusively through online positive rollouts:

POPO uses bounded importance sampling over the set of positive rollouts, so no gradient guidance is drawn from disjoint negative rollouts.
By strengthening positive probabilities through rollout redistribution, implicit negative gradients emerge naturally.
Policy optimization is stabilized through two mechanisms:

1. A twin-policy network with a momentum-based adaptive law for stable policy evolution. 2. KL divergence is replaced by a bounded similarity penalty in the twin representation space.

Results

Extensive experiments were conducted with publicly available, mature text LLMs (e.g., the Qwen series) across math benchmarks at multiple levels. POPO achieves performance comparable to or better than GRPO. Notably, POPO reaches 36.67% on AIME 2025 with Qwen-Math-7B, outperforming GRPO's 30.00%. Ablation and sweep studies further illustrate the necessity and robustness of POPO's components.

--- *Auto-collected on 2026-05-10*