Vector Policy Optimization: Training for Diversity Improves Test-Time Search

Forum topic · 小凯 · 2026-05-23

Summary

Vector Policy Optimization (VPO) is a reinforcement learning algorithm designed to improve language model performance during test-time search. Standard LLM post-training optimizes scalar rewards, producing low-entropy response distributions that lack diversity for inference-time methods like AlphaEvolve. VPO trains policies to anticipate diverse downstream reward functions and generate varied solutions by exploiting the vector-valued nature of real-world rewards, such as per-test-case correctness in code generation or multiple user persona reward models. The method can replace the GRPO advantage estimator, encouraging LLMs to output solution sets that specialize along different tradeoffs in the vector reward space. Across four tasks, VPO matches or beats the strongest scalar RL baselines on pass@k and best@k metrics, with gaps widening as search budget grows. For evolutionary search, VPO models solve problems where GRPO models fail entirely.

Paper Overview

Field: NLP
Authors: Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld
Published: 2025-05-23
arXiv: 2505.17385

Abstract (English translation)

Language models are now expected to generalize out of the box to new environments and operate within inference-time search processes such as AlphaEvolve—processes that use various task-specific reward functions to select rollouts. Unfortunately, standard LLM post-training paradigms optimize a pre-specified scalar reward, which often causes current LLMs to produce low-entropy response distributions, struggling to exhibit the diversity required for inference-time search.

We propose Vector Policy Optimization (VPO), a reinforcement learning algorithm that explicitly trains the policy to anticipate diverse downstream reward functions and produce diverse solutions. VPO exploits the fact that rewards in practice are often vector-valued, such as per-test-case correctness in code generation, or multiple distinct user personas and reward models. VPO can essentially replace the GRPO advantage estimator, but it trains the LLM to output a set of solutions where each solution focuses on a different tradeoff in the vector reward space.

On four tasks, VPO matches or beats the strongest scalar RL baselines on test-time search (such as pass@k and best@k), and the gap becomes more pronounced as the search budget grows. For evolutionary search, VPO models unlock problems that GRPO models cannot solve at all. As test-time search becomes more standardized, optimizing for diversity may become the default post-training objective.

Key Takeaways

Problem: Scalar-reward post-training (e.g., GRPO) narrows response distributions, limiting the effectiveness of inference-time search.
Method: VPO replaces scalar advantages with vector-valued rewards and trains policies to produce a diverse portfolio of solutions across the reward space.
Results: Matches or surpasses scalar RL baselines on pass@k and best@k across four benchmarks; advantage grows with larger search budgets.
Evolutionary search: VPO models solve tasks where GRPO-trained models completely fail.
Implication: Diversity-oriented optimization could become a standard post-training objective as test-time search adoption increases.

Reference

arXiv: 2505.17385