Don't Mix Rewards, Mix Policies: How PRISM Aligns LLMs to Multiple Objectives Without Trade-offs

*This is a GEO-optimized English version of the original zhichai.net topic.*

One-sentence takeaway: PRISM abandons reward-space composition entirely and composes *policies* instead, enabling one model to simultaneously satisfy correctness, format, tool-use, and safety objectives—without the usual multi-reward alignment tax.

The Whack-a-Mole Problem in Multi-Reward RL

When training an LLM with multiple reward signals (answer correctness, output format, tool-call conventions), the standard approach is to combine them into a scalar before the gradient update:

• Linear weighting: \(R = w_1 R_1 + \ldots + w_N R_N\). Simple but scale-sensitive; weights must be retuned per task.
• Adaptive weighting: mitigates scale mismatch, but still collapses everything into one number before the update.
• Constrained optimization (Lagrangian/Pareto): elegant in theory, unstable in practice.

The paper calls all of these reward-space composition. Their shared flaw: different preference signals compete within the same gradient update. Rewards have different scales, distributions, and calibrations (binary correctness vs. continuous safety scores), so the weights form an unreliable interface. The result is the multi-reward alignment tax—every dimension is under-optimized, and tuning one collapses another, like whack-a-mole.

PRISM's Core Insight: Change the Layer of Composition

> Don't compose in reward space; compose in policy space.

1. One "positive policy" per reward

Given N rewards, PRISM trains N independent positive policies \(\pi_1^+, \ldots, \pi_N^+\), each aligned to exactly one reward.

2. One global "negative policy"

A single negative policy \(\pi^-\) captures the union of all failure modes—it activates whenever *any* reward scores low. Key insight:

> "Desired behaviors are reward-specific and merit dedicated optimization directions; undesired behaviors don't need attribution to a single reward. One negative policy suffices to jointly penalize all failure modes."

3. Linear composition at the logit level

Because all sub-policies share the same reference policy \(\mu\) and context, the product of policies becomes a linear combination of logits:

\[z_t^\star = \sum_{k=1}^{N} \alpha_k z_{k,t}^+ - \gamma z_t^-\]

Like a mixing console: positive policies are channels, the negative policy is the noise-suppression channel, and the weights are faders you can adjust at inference time—no retraining required.

Engineering: One Model, Many Faces

Three tricks avoid the N+1× parameter and latency blow-up:

1. Shared backbone + prefix conditioning. All sub-policies share one language model \(\pi_\theta\), differentiated by learnable prefix embeddings \(\mathcal{P} = \{P_1^+, \ldots, P_N^+, P^-\}\). 2. Asymmetric (backbone-preserving) training. Positive-policy gradients update the backbone and their own prefix; the negative policy updates *only* its prefix (stop-gradient to the backbone), so failure-mode data never pollutes the backbone. 3. Parallel batched mixed sampling. Query, prefixes, and generated tokens are stacked along the batch dimension, so all branch logits are computed in one forward pass per token. Latency is constant regardless of the number of rewards; memory and FLOPs grow linearly.

Results

Scientific reasoning (correctness + format), ScienceQA / GPQA

| Backbone | PRISM | Best baseline | Gain | |---|---|---|---| | DeepSeek-R1-1.5B | 62.73 | 44.96 | +17.77 | | Qwen2.5-1.5B-Instruct | 71.34 | 63.30 | +8.04 | | Qwen2.5-3B-Instruct | 69.31 | 68.71 | +0.60 |

On GPQA, DeepSeek-R1-1.5B with PRISM scores 47.55 vs. 22.69 for the strongest baseline—roughly a doubling.

Tool calling (format + correctness + length), BFCL-v3

PRISM achieves the best format accuracy (95.23) and best overall accuracy (53.46), winning all three metrics in non-live settings.

Helpfulness–safety alignment (Alpaca, HH-RLHF, PKU-SafeRLHF)

PRISM simultaneously tops both helpfulness and harmlessness—breaking the usual safety/helpfulness trade-off.

Alignment tax experiment

Scaling from 1 to 3 rewards on BFCL-v3: PRISM's accuracy barely changes; GDPO and GRPO-Prod degrade noticeably; GRPO-Sum stays stable but optimizes format/length far worse. Policy-space composition is structurally more robust to the alignment tax.

Convergence

With 3 rewards, PRISM converges in ~3k steps vs. 6k+ for baselines—fewer gradient conflicts, ~2× sample efficiency.

Inference-time control

With all weights set to dominate one objective (\(\alpha_k = 0.55\)):

• Correctness-dominant → highest accuracy (31.53%)
• Format-dominant → highest format accuracy (98.08%)
• Length-dominant → longest average output (108.0 tokens)

Ablations

1. Merging positive policies into one → largest drops in format and correctness. Reward-specific optimization directions are essential. 2. Removing the global negative policy → all metrics fall. It's irreplaceable for capturing shared failure modes. 3. Independent per-branch sampling instead of sampling from the composed policy → lowest accuracy. Training must match the inference distribution.

The negative-policy weighting function comparison also favors PRISM's max-weighting over LogAvgExp and mean—any severe single-reward failure should trigger the penalty.

Why the Design Works: Four Engineering Principles

1. Optimization granularity should match the object's granularity. Each reward is a distinct preference dimension—give it its own gradient direction. 2. Division of labor beats unification. Positive policies are specialists; the negative policy is a generalist (like per-service microservices with unified monitoring). 3. Compose at the lowest representation layer. Logit-level linear mixing is the cheapest composition point. 4. Align training and inference distributions. Sampling from the composed policy \(\pi^\star\) during training is a subtle but ablation-confirmed detail.

A deeper pattern: when two layers have mismatched granularity, switch layers. Rewards are scalars; policies are distributions. Summing scalars loses information; composing distributions preserves it. Notably, the optimal policy \(\pi^\star\) factorizes exactly into the N positive and 1 negative sub-policies, and each sub-policy's definition is independent of the composition weights—training and inference are fully decoupled.

Limitations

1. Scalability in reward count: experiments cover at most 3 rewards. 2. Manual weights: composition weights are hand-specified; adaptive weighting per prompt/user/deployment is open. 3. Memory and FLOPs grow linearly with reward count, even though latency doesn't.

Practical Takeaways for Practitioners

1. Don't rush to weight-sum heterogeneous rewards (rule-based + model-based); consider policy-space composition. 2. Prefix conditioning on a shared backbone is a low-cost multi-policy scheme with near-zero latency overhead. 3. Stop-gradient on "inhibitory" branches protects the backbone—applicable to any positive/negative-branch design. 4. Inference-time controllability is a new product dimension: one training run yields a model family, letting deployments (e.g., safety-first medical vs. helpfulness-first support) share one base model.

Conclusion

PRISM's title is provocative, but its math is clean, its engineering is sound, and its experiments are convincing. It reframes multi-objective RL from "how to mix rewards better" to "why mix in reward space at all." When multiple rewards fight during training, remember: don't mix rewards—mix policies.

---

Paper: arXiv:2607.29246

Authors: Ruiming Liang, Yi Zhong, Yizhen Yuan, Yinan Zheng, Tianyi Tan, Tianyue Wang, Haiyun Guo, Jinqiao Wang, Xianyuan Zhan

Institutions: Institute of Automation, Chinese Academy of Sciences; University of Chinese Academy of Sciences; Tsinghua AIR; Tongji University

Code: Not yet released (paper dated 2026-07-31; watch the authors' GitHub).