GD2PO: A Signal Denoiser for Multi-Reward Reinforcement Learning

Forum topic · 小凯 · 2026-06-17

Summary

GD2PO (Group-Dynamic reward-Decoupled Policy Optimization) is a lightweight extension of GDPO, proposed by a team including Alibaba's Qwen group, that mitigates multi-reward conflicts in LLM post-training. When useful, safe, format, and length reward models are aggregated via weighted advantage sums, opposing signals cancel out and the policy learns little. GD2PO adds a two-layer denoising architecture: (1) rollout-level conflict filtering via hard sign-consistency checks or a signal-to-noise ratio (SNR) criterion that keeps rollouts whose per-dimension advantages constructively interfere, and (2) query-level dynamic reweighting, scaling updates by the fraction of rollouts retained per group, justified by a variance analysis showing effective SNR grows as sqrt(kappa). On tool calling (API-Bank, Qwen2.5-3B) and helpfulness-harmlessness alignment (HH-RLHF / PKU-SafeRLHF / Alpaca, Qwen2.5-7B), GD2PO improves accuracy and overall scores over GRPO and GDPO without sacrificing auxiliary rewards, at near-zero computational overhead. Code is available at https://github.com/Qwen-Applications/GD2PO.

Overview

Paper: GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization Authors: Haotian Liu, Yihao Liu, Jingwei Ni, et al. (Alibaba Qwen team + Renmin University / Peking University / ETH / CUHK) arXiv: 2606.16771 Code: https://github.com/Qwen-Applications/GD2PO

The Core Problem: Multiple Rewards = Multiple Noise Sources

LLM post-training is shifting from single rewards to multi-dimensional rewards — helpfulness, safety, conciseness, instruction following, tool calling — each with its own reward model. The hidden trap: reward dimensions often point in opposite directions. A response may score high on helpfulness but low on safety. When GDPO aggregates per-dimension advantages via weighted sums, positive and negative signals cancel, and the aggregate advantage approaches zero — the model learns nothing.

This is the multi-reward conflict problem.

GDPO's Limitation: The Cost of Late Aggregation

GDPO improves on GRPO via decoupled normalization — each reward dimension is independently group-normalized before aggregation:

A_n^{\text{sum}} = \sum_{i=1}^{M} w_i A_n^{i}

But this is still late aggregation: at the rollout level, per-dimension advantages are already merged into a scalar. Experiments show a significant conflict ratio during training, varying by task and model.

GD2PO's Two-Layer Denoising Architecture

Core philosophy: identify and intercept cross-reward conflicts before final aggregation.

Layer 1: Rollout-Level Conflict Filtering

Hard filtering: discard a rollout if any two reward dimensions have opposite advantage signs:

\delta_{\text{hard}}(y_n) = \mathbf{1}\{\text{Sign}(A_n^{i}) = \text{Sign}(A_n^{j}), \forall i,j\}

SNR-based soft filtering: quantify conflict via a signal-to-noise ratio:

\text{SNR}_n = \frac{|\sum_{i=1}^{M} w_i A_n^{i}|}{\sum_{i=1}^{M} |w_i A_n^{i}| + \epsilon}

SNR → 1 means constructive interference; SNR → 0 means severe cancellation. Keep rollouts with SNR > τ (0.5–0.8 in experiments).

Layer 2: Query-Level Dynamic Reweighting

After filtering, groups retain different numbers of rollouts. GD2PO uses the retention ratio as the update weight:

\hat{\kappa}(x) = \frac{1}{G} \sum_{n=1}^{G} \delta(y_n)

Theoretical analysis shows the effective signal-to-noise ratio of retained gradients scales as $\sqrt{\kappa(x)}$ , so queries with low retention should indeed be downweighted.

Full Objective

\mathcal{J}_{\text{GD}^2\text{PO}} = \mathbb{E}_{x, \mathcal{G}(x)} \left[ \frac{1}{G} \hat{\kappa}(x) \sum_{n=1}^{G} \frac{1}{|y_n|} \sum_{t=1}^{|y_n|} \gamma_n^{t}\left(\theta, \delta(y_n) \cdot \sum_{i=1}^{M} w_i A_n^{i}\right) \right]

Three components: the rollout-level filter mask $\delta(y_n)$ , the query-level retention reweighting $\hat{\kappa}(x)$ , and the standard PPO/GRPO clipped surrogate $\gamma_n^t$ . Compute overhead is near zero — only sign checks and SNR computation, no extra trainable parameters.

Experiments

| Task | Reward Dimensions | Dataset | |------|------------------|---------| | Tool calling (2 rewards) | Correctness + Length | API-Bank | | Tool calling (3 rewards) | Correctness + Format + Length | API-Bank | | Helpfulness-safety alignment | Useful + Harmless | HH-RLHF / PKU-SafeRLHF / Alpaca |

Baselines: GRPO, GDPO, GD2PO-Hard, GD2PO-SNR.

Two-reward tool calling (Qwen2.5-3B):

| Method | Correct Acc. | Length Rew. | Overall | |--------|-------------|------------|---------| | GRPO | 61.24 | 0.99 | 1.602 | | GDPO | 61.57 | 1.00 | 1.616 | | GD2PO-Hard | 62.88 | 1.00 | 1.629 | | GD2PO-SNR | 62.35 | 1.00 | 1.624 |

GD2PO-Hard gains 1.31–1.74 pp on Correct Acc. with no sacrifice to the auxiliary length reward.

Helpfulness-safety alignment (Qwen2.5-7B): Overall averages 5.425 (GRPO), 5.600 (GDPO), 5.703 (GD2PO-Hard), 5.667 (GD2PO-SNR) — both objectives improve simultaneously.

Three-reward setting: GD2PO-SNR overtakes Hard, suggesting sign checks suffice for two rewards while SNR is needed to distinguish mild disagreement from severe conflict with three or more.

Ablations: Filtering alone gives +0.6–0.9 pp; query-level reweighting adds +0.3–0.5 pp. SNR threshold τ is robust across 0.3–0.8, with τ = 0.5 best.

Training Dynamics Insights

Conflicts are ubiquitous across all models and tasks.
Conflict dynamics vary: some drop early, others persist or peak late.
More reward dimensions → more conflicts.
Helpfulness-safety conflicts are the most persistent, reflecting a fundamental tension.

Relation to Prior Work

GD2PO uniquely bridges DAPO's dynamic sampling idea (filtering degenerate groups) and GDPO's multi-reward decoupled normalization, extending "filtering low-quality signals" from single-reward to multi-reward settings.

Limitations

1. SNR filtering depends on threshold τ; adaptive thresholds are a future direction. 2. Filtered rollouts may contain latent useful signals; extreme cases risk over-pruning. 3. GD2PO improves single-point optimization stability, not Pareto front exploration. 4. Sign/SNR-based filtering assumes meaningful advantage directions; other policy spaces may need adaptation.

One-Sentence Takeaway

GD2PO is a signal denoiser for multi-reward RL — intercepting cross-dimension conflicts before advantage aggregation turns noisy weak signals into clean strong ones, with no new architecture and no extra parameters.