Vector Policy Optimization (VPO): Training LLMs with Vector Rewards for Better Test-Time Search

Forum topic · 小凯 · 2026-06-12

Summary

Vector Policy Optimization (VPO), proposed by MIT researchers, addresses a key limitation of scalar-reward RL methods like GRPO: they collapse multi-dimensional rewards into a single scalar, producing models that learn one "average optimal" strategy with low output diversity. VPO instead treats rewards as vectors and optimizes a set-level objective that randomly samples weights from a Dirichlet(1) distribution, encouraging the policy to produce multiple candidate answers covering the Pareto frontier. Models generate several candidates in a single rollout via multi-answer chains, enabling in-context exploration. Evaluated on Maze, MuSiQue, EUREQA, ToolRL, and LiveCodeBench with Qwen3 models, VPO consistently outperforms GRPO under best@k search, with the advantage growing with search budget—for example, VPO at k=3 exceeds GRPO at k=30 on MuSiQue, and GRPO's diversity plateaus near zero. Ablations show 3x compute cannot substitute for VPO's diversity, and goal-conditioned GRPO suffers mode collapse. VPO underperforms only when reward dimensions are nearly collinear. As test-time search becomes standard, VPO suggests diversity-aware RL post-training should be the default.

Vector Policy Optimization (VPO): Training LLMs with Vector Rewards for Better Test-Time Search

VPO, proposed by Ryan Bahlous-Boldi et al. (MIT, Improbable AI Lab, MIT-IBM Computing Research Lab, Sakana AI), challenges a core assumption of GRPO/PPO/DPO-style post-training: that rewards are scalars. The paper argues that when test-time search (best@k, evolutionary search) is standard, RL post-training should focus on producing *diverse, capable* solutions and leave selection to search. (arXiv:2505.17385)

Key points

The problem with scalar RL: Real rewards are multi-dimensional — code tests pass/fail independently, maze navigation involves path length + coins + diamonds + lava avoidance. Scalar RL compresses these into one number, yielding a single "average optimal" strategy. When search needs diversity, the model can't provide it.
Core insight: The larger the search budget, the more diversity is worth. Training handles exploration (covering the reward space); inference-time search handles exploitation.

The VPO algorithm

Rewards are vectors r ∈ R^d. The set-level objective randomly samples weight vectors w ~ Dir(1) covering the entire simplex and maximizes E_w[max_{y∈S} w^T r(x, y)], so whatever weight the future search uses, the candidate set contains a strong answer.
Multi-answer chains: the model emits ~3 candidates in one autoregressive rollout, separated by delimiters, enabling *in-context exploration* — later candidates can avoid already-covered regions. No external sampling needed.
VPO is a drop-in replacement for GRPO: shared sampled weights per group, Monte-Carlo set rewards, standard z-score token-level advantages, zero architecture changes.

Results (Qwen3-1.7B/4B/8B, Qwen2.5-Coder-7B)

MuSiQue (multi-hop QA): GRPO barely improves with search (0.711 at k=3 → 0.728 at k=30). VPO goes 0.742 → 0.832, so VPO@k=3 already beats GRPO@k=30. Diversity: 0.587 vs 0.054 (~10x).
Maze: GRPO completely plateaus at 0.432 (diversity 0.003); VPO reaches 0.593 at k=30 (diversity 1.006) — nearly double.
EUREQA (causal-chain entity extraction): VPO grows steadily 0.213 → 0.279 vs GRPO's 0.212 → 0.236.
ToolRL: near ceiling for all methods, but VPO catches up by k=10 with diversity 1.297 vs 0.044.
LiveCodeBench (hardest 32 problems): with OpenEvolve evolutionary search, VPO-trained models keep discovering new solutions and crack some problems; GRPO plateaus immediately.

Ablations

Giving GRPO 3x compute still underperforms VPO — scalar search cannot replace trained-in diversity.
Goal-conditioned GRPO (explicitly conditioning on sampled w) suffers mode collapse; in-context exploration works better.

Limitations

VPO fails when reward dimensions are nearly collinear (UltraFeedback + ArmoRM-5): the simplex collapses to a line and VPO underperforms the scalar baseline. True multi-objective reward structure is required.

Conclusion

Scalar RL trains models to answer "what is the best answer"; vector RL trains them to answer "a set of answers covering every definition of good." As test-time search (AlphaEvolve, Best-of-N, MCTS) becomes standard, diversity-aware training shifts from optional to default.

Reference: Bahlous-Boldi, R., Puri, I., Shenfeld, I., et al., "Vector Policy Optimization: Training for Diversity Improves Test-Time Search", 2025.

Vector Policy Optimization (VPO): Training LLMs with Vector Rewards for Better Test-Time Search

Summary

Vector Policy Optimization (VPO): Training LLMs with Vector Rewards for Better Test-Time Search

Key points

The VPO algorithm

Results (Qwen3-1.7B/4B/8B, Qwen2.5-Coder-7B)

Ablations

Limitations

Conclusion

Tags