Vector Policy Optimization: Why the "Best" AI Models Collapse—and How Vector Rewards Fix It

Forum topic · 小凯 · 2026-05-23

Summary

This forum post analyzes Vector Policy Optimization (VPO), a reinforcement learning method (arXiv:2605.22817) that trains LLMs for diversity instead of a single scalar optimum. Standard RL post-training with scalar rewards (e.g., GRPO) causes policy collapse: the model converges to a narrow set of high-probability answers, destroying the diversity required for test-time search like Best-of-N, MCTS, or evolutionary search. VPO keeps rewards as vectors (e.g., per-test-case correctness, multi-hop step accuracy, tool-usage sub-scores) and optimizes a set-level objective: the expected maximum reward over a generated candidate set under random Dirichlet-sampled weight preferences, encouraging Pareto frontier coverage. It combines multi-answer chains with stochastic scalarization; ablations show either mechanism alone, extra compute, per-dimension normalization, or goal-conditioning fails to match it. Across MuSiQue, a Maze benchmark, EUREQA, ToolRL, and LiveCodeBench, VPO's advantage grows with search budget k, while GRPO saturates quickly. VPO sacrifices some pass@1 performance and does not help when reward dimensions are collinear, rewards are purely scalar, or tasks are too easy. The post argues diversity should become a default post-training objective as test-time search becomes standard in AI systems.

Vector Policy Optimization: Training for Diversity Improves Test-Time Search

> Paper: Vector Policy Optimization: Training for Diversity Improves Test-Time Search > Authors: Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld, Akarsh Kumar, Mehul Damani, Sebastian Risi, Omar Khattab, Zhang-Wei Hong, Pulkit Agrawal > arXiv: 2605.22817

Key points

The problem: LLMs are increasingly embedded in test-time search pipelines (AlphaEvolve, Best-of-N sampling, MCTS) that require a diverse pool of high-quality candidates. But scalar-reward RL post-training (e.g., GRPO) causes *policy collapse*: the policy converges to a narrow set of high-probability responses, and additional samples become near-duplicates.
Core insight: Real rewards are naturally *vectors*, not scalars — per-test-case correctness, multi-hop step accuracy, helpfulness/harmlessness dimensions, or tool-usage sub-scores (format, tool name, parameter keys, values).
VPO objective: A set-level reward covering the Pareto frontier:

R(S) = \mathbb{E}_{w \sim \text{Dir}(\alpha)} \left[ \max_{y \in S} w^\top r(x,y) \right]

A good candidate set $$S$$ should contain at least one strong answer under any possible weight preference $$w$$ .

Two synergistic mechanisms:

\(w\)

VPO vs. GRPO

| Dimension | GRPO | VPO | |:---|:---|:---| | Output | Single answer $$y$$ | Set $S = \{y_1, ..., y_m\}$ | | Reward | Scalar | Vector $r(x,y) \in \mathbb{R}^d$ | | Objective | Max expected scalar reward | $\mathbb{E}_w[\max_{y\in S} w^\top r]$ | | Training distribution | Fixed weights | Random $w \sim \text{Dir}(\mathbf{1})$ | | Equilibrium | Single-mode collapse | Pareto frontier coverage |

Experimental results

The advantage of VPO grows with the test-time search budget $$k$$ :

MuSiQue (multi-hop QA): GRPO improves only 1.7% from best@3 to best@30 (0.711→0.728); VPO improves 9.0% (0.742→0.832), widening the gap to 10.4%.
Maze (forced trade-offs): GRPO shows *zero* improvement across k (0.432 everywhere) with near-total collapse (diversity 0.003 vs. VPO's 1.006 — a 335x gap). VPO wins even when evaluated with GRPO's own scalar reward.
EUREQA (equation discovery): VPO leads at all k (best@30: 0.279 vs. 0.236).
ToolRL: VPO reaches best@30 of 0.952 vs. GRPO's 0.925, with 29x more diversity.
LiveCodeBench: At k=1, GRPO is better; in the search regime (k>1) and long evolutionary search, VPO immediately overtakes and keeps finding new solutions.

Ablations

Multi-answer alone (Multi-RLVR) still collapses — capacity without incentive.
Random-weight GRPO alone fails — single answers can't cover trade-off points; weight fluctuation destabilizes training.
Extra compute doesn't help GRPO: 3x rollouts (n=24, best@3=0.763) still loses to VPO with 1/3 the compute (n=8, best@3=0.779).
Per-dimension normalization (GDPO) ≈ GRPO — the gain comes from the objective structure, not gradient conditioning.
Goal-conditioned GRPO fails catastrophically (Maze best@3: 0.205 vs. 0.512): explicitly encoding preferences as text conditions does not reliably translate into behavior; VPO's implicit random-weight training is more effective.

When VPO does not help

1. Highly collinear reward dimensions (the Dirichlet simplex degenerates). 2. Purely scalar rewards with no natural decomposition. 3. k=1 evaluation — VPO trades pass@1 for search-time diversity. 4. Tasks too easy for search to matter.

As the paper states: "VPO is for the regime where test-time search is part of the system."

Outlook

The authors predict that "as test-time search becomes more standardized, optimizing for diversity may need to become the default post-training objective." VPO represents a paradigm shift: from training one optimal student to training a collectively strong team that covers the space of possible trade-offs — acknowledging that open problems rarely have a single correct answer.