An Awkward Reality
You train an LLM to be correct, well-formatted, and good at tool use. The standard approach: sum three rewards into one scalar and feed it to GRPO. The result — every dimension improves a little, none reaches its best, and adjusting one weight breaks another. Whack-a-mole. This is not a hyperparameter problem; it is a structural flaw in reward-space composition.
On July 31, 2026, a team from the Institute of Automation (CAS), UCAS, Tsinghua AIR, and Tongji University proposed PRISM, with a provocative title:
> Don't Mix Rewards, Mix Policies.
The Multi-Reward "Whack-a-Mole" Dilemma
Existing methods (linear weighting, adaptive weighting, constrained optimization) all perform reward-space composition: collapse N rewards \(R_1, \ldots, R_N\) into one scalar before the gradient update. Different preference signals compete inside the same gradient step — the author's term is the multi-reward alignment tax. Reward weights are an unreliable interface: signals differ in scale, distribution, and calibration, so one knob controls volume, brightness, and temperature simultaneously.
PRISM's Core Insight: Compose in Policy Space
1. One "positive policy" per reward
Instead of one policy on a merged reward, PRISM trains N independent positive policies \(\pi_1^+, \pi_2^+, \pi_3^+\), each targeting one reward.
2. One global "negative policy" for all failure modes
A single negative policy \(\pi^-\) captures the union of failure modes across all rewards. Key insight from the paper: ideal behavior is reward-specific and deserves dedicated optimization directions; undesired behavior does not need attribution to a single reward.
3. Linear composition at the logit layer
Because all sub-policies share the same reference policy \(\mu\) and context, the product of policies becomes a linear combination of logits:
Like a mixing console: each positive policy is a channel, the negative policy a noise-reduction channel, and weights are faders that can be adjusted at inference — no retraining needed.
One Model, Many Faces
Three engineering tricks avoid N+1 separate models:
1. Shared backbone + prefix conditioning. One LM \(\pi_\theta\) with independent learnable prefix embeddings per sub-policy. 2. Asymmetric training. Positive-policy gradients update both backbone and prefix; the negative policy updates only its own prefix (stop-gradient to backbone) — the backbone is never polluted by failure modes ("backbone-preserving asymmetric training"). 3. Parallel batched mixing sampling. Query, prefixes, and generated tokens are stacked along the batch dimension, so all branch logits are computed in one forward pass. Per-token latency equals a single pass; memory and FLOPs grow linearly with the number of rewards, but latency does not.
The Numbers
Scientific reasoning (correctness + format), ScienceQA / GPQA
| Backbone | PRISM | Strongest baseline | Gain | |------|-------|---------|------| | DeepSeek-R1-1.5B | 62.73 | 44.96 | +17.77 | | Qwen2.5-1.5B-Instruct | 71.34 | 63.30 | +8.04 | | Qwen2.5-3B-Instruct | 69.31 | 68.71 | +0.60 |
On GPQA, PRISM with DeepSeek-R1-1.5B reaches 47.55 vs 22.69 for the strongest baseline — roughly doubled.
Tool calling (format + correctness + length)
On BFCL-v3, PRISM achieves best format accuracy (95.23) and best overall accuracy (53.46); best on all three metrics in non-live settings.
Helpfulness–safety alignment
On Alpaca, HH-RLHF, and PKU-SafeRLHF, PRISM scores highest on helpfulness and harmlessness simultaneously — breaking the usual trade-off.
Alignment tax experiment
Training with 1, 2, then 3 rewards on BFCL-v3: PRISM's accuracy barely changes; GDPO and GRPO-Prod drop noticeably; GRPO-Sum stays stable but optimizes format/length much more weakly.
Convergence speed
~3k steps to converge vs 6k+ for baselines in the 3-reward setting — fewer gradient conflicts.
Inference-Time Control
One training run yields a family of policies: shifting mixing weights toward correctness, format, or length yields the best accuracy (31.53%), best format accuracy (98.08%), or longest average reasoning (108.0 tokens) respectively — no retraining.
Ablations
1. Merging positive policies → largest drops in format and accuracy; reward-specific optimization directions are essential. 2. Removing the global negative policy → all metrics degrade. 3. Sampling from branches independently instead of the composed policy → worst overall accuracy; sampling from \(\pi^\star\) aligns training and inference distributions.
Max-weighted negative weights beat LogAvgExp and mean: any severe single-reward failure should trigger punishment.
Engineering Principles
- Optimization granularity should match the object being optimized.
- Division of labor beats unification — specialists for rewards, a generalist for failure modes.
- Compose at the lowest-level interface — logits, not sampled outputs.
- Align training and inference distributions.
Limitations
1. Tested with at most 3 rewards; scaling behavior unknown. 2. Mixing weights are manually specified; adaptive weighting is open. 3. Memory and FLOPs grow linearly with reward count.
Takeaways for Practitioners
1. Don't rush to weight-sum multiple rewards of different natures; consider policy-space composition. 2. Prefix conditioning is a low-cost multi-policy scheme with near-zero latency overhead. 3. Asymmetric training with stop-gradient on the backbone protects it from negative data — applicable to any positive/negative branch design. 4. Inference-time controllability is a new product dimension: train once, deploy a family of policies tuned per scenario (safety-leaning medical, helpfulness-leaning support).
Conclusion
PRISM's title is provocative, but its math is clean, its engineering solid, and its experiments convincing. When multiple rewards fight in training, remember: don't mix rewards, mix policies.
---
Paper: arXiv:2607.29246
Authors: Ruiming Liang, Yi Zhong, Yizhen Yuan, Yinan Zheng, Tianyi Tan, Tianyue Wang, Haiyun Guo, Jinqiao Wang, Xianyuan Zhan
Institutions: Institute of Automation, CAS / UCAS / Tsinghua AIR / Tongji University
Code: not yet open-sourced (paper dated 2026-07-31)