FERPO: Forward Entropy-Regularized Policy Optimization — Critic-Gradient-Free On-Policy RL

Forum topic · 小凯 · 2026-10-04

Summary

FERPO (Forward Entropy-Regularized Policy Optimization) is a new on-policy maximum-entropy reinforcement learning algorithm for continuous control, proposed by Sebastian Sanokowski, Alireza Sarmadi, and Majid Khadiv (arXiv:2610.02198, October 2026). Unlike state-of-the-art methods that improve policies using action gradients of a learned critic, FERPO performs policy improvement using critic values only, without differentiating the critic with respect to actions. This addresses the issue that accurate value predictions do not necessarily yield accurate action derivatives, which can cause unreliable policy updates. FERPO derives an optimal target action distribution from an entropy- and KL-regularized policy-improvement objective, then fits the actor to this target by minimizing a forward-KL objective, estimated with self-normalized importance sampling (SNIS) using rollout-policy actions. KL regularization keeps the target close to the rollout policy so importance weights remain well behaved, while the forward-KL objective—unlike mode-seeking reverse KL—encourages covering multiple high-value modes and promotes exploration. Experiments on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains, and computational benchmarks show faster actor updates than REPPO.

Paper Overview

Research area: Machine Learning

Authors: Sebastian Sanokowski, Alireza Sarmadi, Majid Khadiv

Published: 2026-10-01

arXiv: 2610.02198

Abstract

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates.

FERPO (Forward Entropy-Regularized Policy Optimization) is an on-policy maximum-entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions.

How it works

FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and KL divergence.
The actor is then fit to this target distribution by minimizing a forward-KL objective, estimated with self-normalized importance sampling (SNIS) using actions sampled by the rollout policy.
KL regularization limits how far the target distribution can deviate from the rollout policy, keeping the importance weights well behaved.
Unlike reverse-KL objectives, which can be biased toward a single mode of the target distribution, the forward-KL objective encourages coverage of multiple high-value modes, thereby promoting exploration.

Results

Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains.
Computational benchmarks show that FERPO's actor update is faster than that of REPPO.