Target Policy Optimization (TPO): Decoupling Reward Assignment from Policy Updates in RLHF

Forum topic · 小凯 · 2026-04-09

Summary

Target Policy Optimization (TPO), introduced by Jean Kaddour (arXiv:2504.06257, April 2025), addresses a core problem in reinforcement learning for language models: standard policy gradient methods conflate two decisions—which sampled completions should gain probability mass, and how parameters should move to realize that change—leading to overshooting or undershooting depending on learning rate, clipping, and other optimizer choices. TPO decouples these questions. Given scored completions, it constructs a target distribution q_i proportional to p_i^old * exp(u_i) and fits the policy to that distribution via cross-entropy. The gradient of the loss on sampled completion logits is p^theta - q, which vanishes once the policy matches the target. Experiments on tabular bandits, transformer sequence tasks, and billion-parameter LLMs trained with RLVR show that TPO matches policy gradient baselines such as PG, PPO, GRPO, and DG on easy tasks while substantially outperforming them under sparse rewards. Code is available at https://github.com/JeanKaddour/tpo.

Overview

Field: Machine Learning Author: Jean Kaddour Published: 2025-04-08 arXiv: 2504.06257

Abstract (translated)

In RL, given a prompt, we sample a group of completions from a model and score them. Two questions follow: which completions should gain probability mass, and how should the parameters move to realize that change? Standard policy gradient methods answer both questions simultaneously, so updates may overshoot or undershoot depending on the learning rate, clipping, and other optimizer choices. This paper introduces Target Policy Optimization (TPO), which separates the two questions. Given scored completions, TPO constructs a target distribution q_i ∝ p_i^old exp(u_i) and fits the policy to this distribution via cross-entropy. The gradient of the loss with respect to sampled completion logits is p^θ − q, which vanishes once the policy matches the target. On tabular bandits, transformer sequence tasks, and billion-parameter LLM RLVR, TPO matches PG, PPO, GRPO, and DG on easy tasks, and substantially outperforms them under sparse rewards. Code is available at https://github.com/JeanKaddour/tpo.

Original Abstract

---

*Auto-collected on 2026-04-09.*