GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic Reward-Decoupled Policy Optimization

> Paper: GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization > Authors: Haotian Liu, Yihao Liu, Jingwei Ni, et al. (Alibaba Qwen team + RUC/PKU/ETH/CUHK) > arXiv: 2606.16771 > Code: https://github.com/Qwen-Applications/GD2PO

The Core Problem: Multiple Rewards = Multiple Noise Sources

LLM post-training is shifting from single rewards to multi-dimensional rewards — helpfulness, safety, conciseness, instruction following, tool calling — with a separate reward model scoring each dimension.

But there's a hidden trap: reward dimensions often conflict in direction.

A response may score high on helpfulness (giving the user what they asked for) but low on safety (leaking sensitive information). When GDPO sums per-dimension advantages with weights, positive and negative signals cancel, and the aggregated advantage approaches zero — the model learns nothing.

This is the multi-reward conflict problem.

GDPO's Limitation: The Cost of Late Aggregation

GDPO's improvement over GRPO is decoupled normalization — each reward dimension gets independent group-relative normalization before aggregation:

This is still late aggregation: at the rollout level, per-dimension advantages have already been collapsed into a scalar. If \(A_n^{\text{useful}} > 0\) while \(A_n^{\text{safe}} < 0\), they fight each other in the sum.

Experiments confirm significant conflict ratios during training, varying by task and model — some concentrated early, some persisting throughout training.

GD2PO's Two-Layer Denoising Architecture

Core philosophy: identify and intercept cross-reward conflicts before final aggregation.

Layer 1: Rollout-Level Conflict Filtering

Hard filtering — drop a rollout if any two reward dimensions have opposite advantage signs:

• Pros: zero tolerance, keeps only fully consistent signals
• Cons: may be too aggressive, discarding samples with slight disagreements but clear overall direction

SNR-based soft filtering — quantify conflict severity using signal-to-noise ratio:

\[\text{SNR}_n = \frac{|\sum_{i=1}^{M} w_i A_n^{i}|}{\sum_{i=1}^{M} |w_i A_n^{i}| + \epsilon}\]
• SNR → 1: dimensions aligned (constructive interference)
• SNR → 0: severe cancellation (destructive interference)

Keep condition: \(\text{SNR}_n > \tau\) (threshold set to 0.5–0.8 in experiments).

Layer 2: Query-Level Dynamic Reweighting

After filtering, queries retain different numbers of rollouts. GD2PO uses the survival ratio as the query's update weight:

\[\hat{\kappa}(x) = \frac{1}{G} \sum_{n=1}^{G} \delta(y_n)\]

Lower survival ratio → smaller (more conservative) updates; higher ratio → larger updates. Theoretically, the signal-to-noise ratio of filtered gradients grows as \(\sqrt{\kappa(x)}\):

\[\frac{|\mathbb{E}[g_\delta(x)]|}{\sqrt{\text{Var}[g_\delta(x)]}} \approx \sqrt{\kappa(x)} \frac{|\mu_x|}{\sigma_x}\]

Full Objective

\[\mathcal{J}_{\text{GD}^2\text{PO}} = \mathbb{E}_{x, \mathcal{G}(x)} \left[ \frac{1}{G} \hat{\kappa}(x) \sum_{n=1}^{G} \frac{1}{|y_n|} \sum_{t=1}^{|y_n|} \gamma_n^{t}\left(\theta, \delta(y_n) \cdot \sum_{i=1}^{M} w_i A_n^{i}\right) \right]\]

Three key components: 1. \(\delta(y_n)\) — rollout-level filtering mask 2. \(\hat{\kappa}(x)\) — query-level survival-ratio reweighting 3. \(\gamma_n^t\) — standard PPO/GRPO clipped surrogate

Computational overhead: nearly zero — just sign checks and SNR computation, no extra trainable parameters.

Experiments

| Task | Reward Dimensions | Dataset | |------|-------------------|---------| | Tool calling (2 rewards) | Correctness + Length | API-Bank | | Tool calling (3 rewards) | Correctness + Format + Length | API-Bank | | Helpfulness-safety alignment | Useful + Harmless | HH-RLHF / PKU-SafeRLHF / Alpaca |

Key Results

Two-reward tool calling (Qwen2.5-3B):

| Method | Correct Acc. | Length Rew. | Overall | |--------|-------------|-------------|---------| | GRPO | 61.24 | 0.99 | 1.602 | | GDPO | 61.57 | 1.00 | 1.616 | | GD2PO-Hard | 62.88 | 1.00 | 1.629 | | GD2PO-SNR | 62.35 | 1.00 | 1.624 |

GD2PO-Hard improves Correct Accuracy by 1.31–1.74 pp while keeping the Length Reward at full marks — no sacrifice of auxiliary objectives.

Helpfulness-safety alignment (Qwen2.5-7B): GD2PO-Hard reaches 5.703 vs GRPO's 5.425 and GDPO's 5.600 — helpfulness and safety improve simultaneously.

Three-reward tool calling: GD2PO-SNR overtakes Hard when rewards increase to three dimensions, suggesting sign checks suffice for two rewards while SNR is needed to distinguish mild disagreement from severe conflict at three or more.

Ablations

• Filtering alone gives +0.6–0.9 pp; query reweighting adds another +0.3–0.5 pp. The two are complementary.
• SNR threshold \(\tau\) in 0.3–0.8 outperforms GDPO, with \(\tau = 0.5\) optimal — the method is robust to threshold choice.

Training Dynamics Insights

1. Conflicts are ubiquitous: nonzero conflict ratios across all models and tasks. 2. Dynamic patterns: some models' conflicts drop quickly early; others persist or peak late. 3. More dimensions → more conflicts: three-reward settings show higher early conflict ratios. 4. Helpfulness-safety conflicts are most persistent, reflecting fundamental tension between the two objectives.

Relation to Prior Work

| Method | Level | Strategy | |--------|-------|----------| | Reward re-weighting | Global | Fixed/dynamic reward weight adjustment | | Gradient coordination | Gradient | Align/balance per-dimension gradients | | GDPO | Advantage | Decoupled normalization, but late aggregation | | GD2PO | Rollout + Query | Filter conflicting samples + dynamic reweighting | | DAPO | Group | Filters all-correct/all-wrong query groups |

GD2PO bridges DAPO's dynamic sampling idea and GDPO's multi-reward decoupling, generalizing "filter low-quality signals" from single-reward to multi-reward settings.

Limitations & Thoughts

1. Threshold hyperparameter: SNR filtering depends on \(\tau\); robust experimentally, but the optimum varies by task. Adaptive thresholds are a future direction. 2. Cost of filtering: filtered rollouts may contain latent useful signal; extreme cases risk over-pruning. 3. Relation to Pareto optimization: GD2PO improves single-point optimization stability, not Pareto front discovery; explicit trade-offs may need combination with Pareto methods. 4. LLM specificity: conflict filtering assumes meaningful advantage directions; continuous-control settings may require adaptation.

One-Sentence Takeaway

> GD2PO is a "signal denoiser" for multi-reward RL — intercepting cross-dimension conflicts before advantage aggregation, letting the model learn from clean strong signals instead of weak signals buried in noise.

It needs no new architecture and no extra parameters — just two lightweight checks added to the GDPO pipeline, yet they raise both training efficiency and final quality for multi-reward alignment.

References

• Liu, H., Liu, Y., Ni, J., et al. (2026). GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization. arXiv:2606.16771.