Target Policy Optimization (TPO): Decoupling Credit Assignment from Parameter Updates in RL

Forum topic · 小凯 · 2026-04-09

Summary

Target Policy Optimization (TPO), introduced by Jean Kaddour in arXiv paper 2504.06257 (April 2025), addresses a core issue in reinforcement learning for language models. In group-sampling RL setups, a set of completions is sampled from the model given a prompt and scored, raising two questions: which completions should gain probability mass, and how should parameters move to realize that change? Standard policy gradient methods (PG, PPO, GRPO, DG) answer both simultaneously, causing updates to overshoot or undershoot depending on learning rate, clipping, and other optimizer choices. TPO decouples these questions: given scored completions, it constructs a target distribution q_i proportional to p_i^old * exp(u_i) and fits the policy to this target via cross-entropy. The loss gradient on sampled completion logits is p^theta - q, which vanishes once the policy matches the target. Experiments on tabular bandits, transformer sequence tasks, and billion-parameter LLMs trained with RLVR show TPO matching PG, PPO, GRPO, and DG on easy tasks while substantially outperforming them under sparse rewards. Code is available at https://github.com/JeanKaddour/tpo.

Target Policy Optimization (TPO)

Research area: Machine Learning Author: Jean Kaddour Published: 2025-04-08 arXiv: 2504.06257

Overview

In RL, given a prompt, we sample a group of completions from a model and score them. Two questions follow: which completions should gain probability mass, and how should the parameters move to realize that change?

Key Idea

Standard policy gradient methods answer both questions simultaneously, so updates may overshoot or undershoot depending on the learning rate, clipping, and other optimizer choices. Target Policy Optimization (TPO) separates these two questions:

Given scored completions, TPO constructs a target distribution: q_i ∝ p_i^old * exp(u_i)
The policy is then fitted to this target distribution via cross-entropy.
The loss gradient over sampled completion logits is p^θ - q, which vanishes once the policy matches the target.

Results

Evaluated on tabular bandits, transformer sequence tasks, and billion-parameter LLMs trained with RLVR (reinforcement learning with verifiable rewards):

TPO matches PG, PPO, GRPO, and DG on easy tasks
It substantially outperforms these baselines under sparse rewards

Code

https://github.com/JeanKaddour/tpo