Why the 'Best' AI Can Be the Dumbest: Vector Policy Optimization (VPO) Explained

Forum topic · 小凯 · 2026-05-23

Summary

This post explains Vector Policy Optimization (VPO), a reinforcement learning method for LLMs that trains models to produce diverse answer sets instead of collapsing onto a single high-reward mode. Standard RL post-training like GRPO uses scalar rewards, pushing policies toward a narrow set of responses—which cripples test-time search methods (Best-of-N, MCTS, evolutionary search) that depend on diverse candidate pools. VPO keeps rewards as vectors (e.g., per-test-case, per-hop correctness) and optimizes a set-level objective: E_w~Dir(α)[max_{y∈S} wᵀr(x,y)], rewarding candidate sets that cover the Pareto frontier across reward trade-offs. It combines multi-answer chains (one generation contains several answers via attention to prior ones) with stochastic scalarization (random Dirichlet weights per iteration); ablations show neither mechanism alone suffices. Across MuSiQue, a multi-objective Maze, EUREQA, ToolRL, and LiveCodeBench, VPO's advantage over GRPO grows with search budget k: e.g., MuSiQue best@30 0.832 vs 0.728, with ~335x diversity on Maze. Known limits: collinear reward dimensions, purely scalar rewards, and single-sample (k=1) evaluation where GRPO wins. The post argues diversity may need to become a default post-training objective as search-driven AI systems become standard.

Why the "Best" AI Can Be the Dumbest: Vector Policy Optimization (VPO)

> Paper: Vector Policy Optimization: Training for Diversity Improves Test-Time Search > Authors: Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld, Akarsh Kumar, Mehul Damani, Sebastian Risi, Omar Khattab, Zhang-Wei Hong, Pulkit Agrawal > arXiv: 2605.22817

The Problem: Scalar Rewards Cause Policy Collapse

LLMs are no longer deployed as standalone components—they are embedded in larger test-time search pipelines: AlphaEvolve's evolutionary code search, Best-of-N sampling, Monte Carlo Tree Search. All of these rely on a diverse pool of high-quality candidates.

But standard RL post-training works against this. With a scalar reward (e.g., correctness 0/1), policy gradient methods like GRPO drive the policy toward a narrow set of high-probability responses. After training, "the diversity required for effective test-time search disappears, as additional samples become near-duplicates." Like a chef optimizing only the easiest-to-score dimensions (taste + plating) while ignoring creativity, nutrition, and speed, the model collapses to one safe template.

The Key Insight: Rewards Are Naturally Vectors

Real rewards are multi-dimensional:

Code generation: pass/fail per test case → $r \in \{0,1\}^d$
Multi-hop reasoning: correctness per hop → $r \in \{0,1\}^d$
RLHF alignment: helpful / harmless / honest / verifiable → $r \in \mathbb{R}^d$
Tool use: format / tool name / argument keys / argument values → $r \in \mathbb{R}^d$

These dimensions involve unavoidable trade-offs—no single solution maximizes all of them.

VPO's Set-Level Objective

VPO trains on sets of answers, not single answers:

R(S) = \mathbb{E}_{w \sim \text{Dir}(\alpha)} \left[ \max_{y \in S} w^\top r(x,y) \right]

A good candidate set $$S$$ should contain at least one strong solution under *any* possible weighting of the reward dimensions—i.e., cover the Pareto frontier.

Two mechanisms must work together:

1. Multi-Answer Chains: one forward pass generates several answers, later ones attending to earlier ones so the model can steer into uncovered regions. 2. Stochastic Scalarization: each training iteration samples a random Dirichlet weight vector defining "what matters this round."

Ablations show neither alone suffices: multi-answer + fixed scalar reward (Multi-RLVR) still collapses; random weights + single answers are unstable. Multi-answer gives *capacity*, random weights give *incentive*.

GRPO vs. VPO

| Dimension | GRPO | VPO | |:---|:---|:---| | Output | single answer $$y$$ | set $S=\{y_1,...,y_m\}$ | | Reward | scalar $w^{*\top}r$ | vector $$r(x,y)$$ , full dimensionality | | Objective | max expected scalar reward | $\mathbb{E}_w[\max_{y\in S} w^\top r]$ | | Gradient effect | all samples pushed to same optimum | different samples rewarded under different weights | | Equilibrium | single-mode collapse | Pareto frontier coverage |

Experimental Results: The Advantage Grows with Search Budget

MuSiQue (multi-hop QA): GRPO improves only 1.7% from best@3 to best@30 (0.711→0.728, diversity 0.054); VPO improves 9.0% (0.742→0.832, diversity 0.587).

Maze (four conflicting objectives): GRPO is completely collapsed—0.432 at every k, diversity 0.003. VPO rises 0.512→0.593 with diversity 1.006 (~335x GRPO's). Even evaluated with GRPO's own uniform scalar, VPO wins: covering the Pareto frontier helps even single-scalar goals.

EUREQA (equation discovery): VPO leads at every k (best@30: 0.279 vs 0.236).

ToolRL: GRPO saturates at 0.925; VPO reaches 0.952 at best@30 with 29x diversity.

LiveCodeBench (Qwen2.5-Coder-7B-Instruct, 279 problems): GRPO wins at k=1; VPO overtakes immediately at k>1 and keeps finding new solutions in 200-round evolutionary search while GRPO plateaus.

Key Ablations

Multi-answer alone? No—Multi-RLVR collapses in reward-space diversity.
Random weights alone? No—Random-Weight GRPO matches or slightly underperforms GRPO.
More compute for GRPO? No—GRPO with 3x rollouts (best@3 = 0.763) still loses to VPO at 1/3 compute (0.779) on MuSiQue.
Per-dimension normalization (GDPO)? No gain—so VPO's benefit comes from the objective structure, not numerical conditioning.
Explicit goal conditioning? Catastrophic—Goal-Conditioned GRPO fails on Maze (best@3 = 0.205 vs 0.512). Text-level conditioning of preferences does not reliably translate into behavior.

When VPO Does Not Help

1. Highly collinear reward dimensions (the Dirichlet simplex degenerates). 2. Naturally scalar rewards—VPO degrades to standard RL. 3. k=1 evaluation—VPO trades pass@1 for search-time diversity. 4. Trivially easy problems where search adds nothing.

As the paper puts it: "VPO is for the regime where test-time search is part of the system."

The Bigger Picture

The paper predicts: "As test-time search becomes more standardized, optimizing for diversity may need to become the default post-training objective." VPO represents a paradigm shift—from training one best student to training a collectively excellent team—recognizing that there is rarely a single correct answer, only different trade-offs under different constraints.

References

Ryan Bahlous-Boldi et al. "Vector Policy Optimization: Training for Diversity Improves Test-Time Search." arXiv:2605.22817.
Compared baselines: GRPO, Multi-RLVR, GDPO, Goal-Conditioned GRPO, Max-at-K, MaxRL
Benchmarks: MuSiQue, Maze, EUREQA, ToolRL, LiveCodeBench, OpenEvolve hardest-32