Vector Policy Optimization: Training for Diversity Improves Test-Time Search

Language models are increasingly the backbone of automated scientific discovery, embedded inside search procedures such as AlphaEvolve that propose candidate programs, equations, or experiments and evolve them against a task-specific objective. Unfortunately, the standard paradigm of LLM post-training optimizes a pre-specified scalar reward, often driving current LLMs toward low-entropy response distributions and leaving them unable to display the diversity that inference-time search requires. We propose Vector Policy Optimization (VPO), an RL algorithm that explicitly trains policies to anticipate diverse downstream reward functions and to produce diverse solutions. VPO exploits the fact that rewards are often vector-valued in practice, such as per-test-case correctness in code generation or per-criterion ratings from multiple reward models. VPO is essentially a drop-in replacement for the GRPO advantage estimator, but it trains the LLM to output a set of solutions in which individual solutions specialize to different trade-offs in the vector reward space. Across four tasks, VPO matches or beats the strongest scalar RL baselines on test-time search (e.g.\ pass@$k$ and best@$k$), with the gap widening as the search budget grows. On evolutionary search, VPO models solve problems the matched GRPO model does not solve within the same search budget. As test-time search becomes more common in discovery pipelines, training for diversity is a useful alternative to scalar reward optimization.