Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-step Policy Gradients

Forum topic · 小凯 · 2026-05-13

Summary

This arXiv paper (2505.07233) by Alex DeWeese and Guannan Qu revisits standard policy gradient methods applied to restricted policy classes, which are known to get stuck in suboptimal critical points. The authors identify a key cause of this phenomenon: the policy gradient is fundamentally myopic, since it improves the policy based only on the one-step Q-function. To address this limitation, they propose a generalized k-step policy gradient method that couples the randomness within a k-step time window, enabling the algorithm to escape myopic local optima that trap standard policy gradient approaches. The work provides a new perspective on why policy gradients fail in constrained policy classes and offers a practical generalization that leverages multi-step temporal dependencies for improved optimization in reinforcement learning.

Paper Overview

Research Area: Machine Learning Authors: Alex DeWeese, Guannan Qu Published: 2025-05-09 arXiv: 2505.07233

Abstract

This work revisits standard policy gradient methods used on restricted policy classes, which are known to get stuck in suboptimal critical points. We identify an important cause for this phenomenon to be that the policy gradient is itself fundamentally myopic, i.e. it only improves the policy based on the one-step Q-function. In this work, we propose a generalized k-step policy gradient method that couples the randomness within a k-step time window and can escape the myopic local optima in...

Key Points

Problem: Standard policy gradient methods on restricted policy classes converge to suboptimal critical points.
Root cause: The policy gradient is fundamentally myopic — it improves the policy based only on the one-step Q-function.
Proposed solution: A generalized k-step policy gradient method that couples randomness across a k-step time window, allowing escape from myopic local optima.

Links: arXiv:2505.07233

--- *Auto-collected on 2026-05-13*