Summary
This forum post summarizes the arXiv paper 2505.17385, 'Vector Policy Optimization' (VPO), by Ryan Bahlous-Boldi, Isha Puri, and Idan Shenfeld, published May 23, 2025 in the NLP domain. Standard LLM post-training optimizes scalar rewards, producing low-entropy response distributions that limit diversity needed for test-time search methods like pass@k, best@k, and evolutionary search (e.g., AlphaEvolve). VPO is a reinforcement learning algorithm that trains policies to anticipate diverse downstream reward functions and produce varied solutions. It exploits the fact that rewards in practice are often vector-valued, such as per-test-case correctness in code generation or multiple user personas and reward models. VPO can directly replace the GRPO advantage estimator, training the LLM to output a set of solutions where each specializes in different trade-offs in vector reward space. Across four tasks, VPO matches or beats the strongest scalar RL baselines on test-time search, with the gap widening at larger search budgets; for evolutionary search, VPO unlocks problems GRPO models cannot solve at all. The authors suggest diversity optimization may become a default post-training objective as test-time search becomes standardized.
Paper Overview
Field: NLP
Authors: Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld
Published: 2025-05-23
arXiv: 2505.17385
Summary
Language models are now expected to generalize out of the box to new environments and to work within inference-time search procedures, such as AlphaEvolve, which use various task-specific reward functions to select rollouts. Unfortunately, the standard LLM post-training paradigm optimizes a pre-specified scalar reward, which often causes current LLMs to produce low-entropy response distributions and struggle to exhibit the diversity required for inference-time search.
The authors propose Vector Policy Optimization (VPO), a reinforcement learning algorithm that explicitly trains the policy to anticipate diverse downstream reward functions and produce diverse solutions. VPO exploits the fact that rewards in practice are often vector-valued — for example, correctness on each test case in code generation, or multiple distinct user personas and reward models.
VPO can natively replace the GRPO advantage estimator, but it trains the LLM to output a set of solutions where each solution specializes in a different trade-off in the vector reward space.
Key Results
- On four tasks, VPO matches or beats the strongest scalar RL baselines on test-time search metrics (pass@k and best@k).
- The performance gap grows as the search budget increases.
- For evolutionary search, VPO-trained models unlock problems that GRPO models cannot solve at all.
The authors conclude that as test-time search becomes more standardized, optimizing for diversity may become the default post-training objective.
---
*Auto-collected on 2026-05-23*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177620658