Global Optimality for Constrained Maximum-Entropy Exploration via Policy Gradient Penalty

Forum topic · 小凯 · 2026-05-04

Summary

This paper proposes the Policy Gradient Penalty (PGP) method for efficient, constrained exploration in reinforcement learning, where exploration is formalized as maximizing the entropy of the state-action occupancy measure. Unlike unconstrained maximum-entropy exploration, real-world exploration must often satisfy safety, resource, or imitation constraints, which is challenging because entropy maximization lacks additive structure, making Bellman-equation-based methods inapplicable, and policy parameterization introduces non-convexity in both objective and constraints. PGP is a single-loop, policy-space method that enforces general convex occupancy-measure constraints through quadratic-penalty regularization. It constructs pseudo-rewards to obtain gradient estimates of the penalized objective, then applies the classical policy gradient theorem. The authors establish smoothness of the penalized objective, supplying the regularity needed for convergence analysis. By exploiting hidden convexity and strong duality, they prove global last-iterate convergence guarantees: PGP achieves an epsilon-optimal constrained entropy value despite the non-convexity induced by policy parameterization, with bounded constraint violation. Authored by Florian Wolf, Ilyas Fatkhullin, and Niao He, the work appears on arXiv (2604.28144) and offers a theoretical foundation for scalable constrained exploration.

Paper Overview

Field: Reinforcement Learning Authors: Florian Wolf, Ilyas Fatkhullin, Niao He Published: 2026-04-30 arXiv: 2604.28144

Abstract (Translated)

Efficient exploration is a central problem in reinforcement learning and is often formalized as maximizing the entropy of the state-action occupancy measure. While unconstrained maximum-entropy exploration is relatively well understood, real-world exploration is often constrained by safety, resource, or imitation requirements.

This constrained setting is particularly challenging because entropy maximization lacks additive structure, rendering Bellman-equation-based methods inapplicable. Moreover, scalable approaches require policy parameterization, inducing non-convexity in both the objective and the constraints.

The authors propose the Policy Gradient Penalty (PGP) method, a single-loop policy-space method that enforces general convex occupancy-measure constraints via quadratic-penalty regularization. PGP constructs pseudo-rewards to produce gradient estimates of the penalized objective, then leverages the classical policy gradient theorem.

The paper further establishes the smoothness of the penalized objective, providing the regularity required to prove PGP's convergence. Exploiting hidden convexity and strong duality, the authors establish global last-iterate convergence guarantees, achieving an epsilon-optimal constrained entropy value under the non-convexity induced by the policy parameterization, with bounded constraint violation.

Key Contributions

A single-loop policy-space method (PGP) for constrained maximum-entropy exploration using quadratic-penalty regularization
Pseudo-reward construction enabling gradient estimation of the penalized objective via the policy gradient theorem
Smoothness analysis of the penalized objective, supplying necessary regularity for convergence proofs
Global last-iterate convergence guarantees with epsilon-optimality and bounded constraint violation, achieved despite policy-induced non-convexity

---

*Auto-collected on 2026-05-04. Source: arXiv:2604.28144*