Paper Overview
Field: Reinforcement Learning Authors: Florian Wolf, Ilyas Fatkhullin, Niao He Published: 2026-04-30 arXiv: 2604.28144
Summary
Efficient exploration is a central problem in reinforcement learning and is often formalized as maximizing the entropy of the state-action occupancy measure. While unconstrained maximum-entropy exploration is relatively well understood, real-world exploration is often constrained by safety, resource, or imitation requirements. This constrained setting is particularly challenging because entropy maximization lacks additive structure, rendering Bellman-equation-based methods inapplicable. Moreover, scalable approaches require policy parameterization, inducing non-convexity in both the objective and the constraints.
The authors propose the Policy Gradient Penalty (PGP) method, a single-loop policy-space method that enforces general convex occupancy-measure constraints via quadratic-penalty regularization. PGP constructs pseudo-rewards to produce gradient estimates of the penalized objective, then leverages the classical policy gradient theorem. The paper further establishes smoothness of the penalized objective, providing the regularity needed to prove PGP's convergence. Exploiting hidden convexity and strong duality, the authors establish global last-iterate convergence guarantees, achieving epsilon-optimal constrained entropy despite policy-induced non-convexity, with bounded constraint violation.
Key Contributions
- PGP method: a single-loop, policy-space algorithm using quadratic-penalty regularization for general convex occupancy-measure constraints
- Pseudo-reward construction enabling policy-gradient-based estimates of the penalized objective
- Smoothness analysis of the penalized objective, supplying required regularity for convergence proofs
- Global last-iterate convergence guarantees via hidden convexity and strong duality, with epsilon-optimal constrained entropy and bounded constraint violation
*Auto-collected on 2026-05-04*