Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-Step Policy Gradients

Forum topic · 小凯 · 2026-05-13

Summary

This arXiv paper (2505.07233) by Alex DeWeese and Guannan Qu revisits standard policy gradient methods applied to restricted policy classes, which are known to get stuck in suboptimal critical points. The authors identify a key cause of this problem: the policy gradient is fundamentally myopic, since it improves the policy based only on the one-step Q-function. To address this limitation, the paper proposes a generalized k-step policy gradient method that couples randomness within a k-step time window, enabling the algorithm to escape myopic local optima that one-step policy gradients cannot overcome. This work contributes to reinforcement learning theory by providing a new perspective on why policy gradient methods converge to suboptimal solutions under restricted policy classes and by offering a principled mechanism to escape such critical points. The full abstract and paper are available on arXiv at https://arxiv.org/abs/2505.07233.

Paper Overview

Field: Machine Learning (ML) Authors: Alex DeWeese, Guannan Qu Published: 2025-05-09 arXiv: 2505.07233

Original Abstract

This work revisits standard policy gradient methods used on restricted policy classes, which are known to get stuck in suboptimal critical points. We identify an important cause for this phenomenon to be that the policy gradient is itself fundamentally myopic, i.e. it only improves the policy based on the one-step Q-function. In this work, we propose a generalized k-step policy gradient method that couples the randomness within a k-step time window and can escape the myopic local optima in restricted policy classes.

Key Points

Problem: Standard policy gradient methods on restricted policy classes can converge to suboptimal critical points.
Root cause identified: The policy gradient is fundamentally *myopic* — it improves the policy based only on the one-step Q-function, ignoring longer-horizon effects.
Proposed solution: A generalized k-step policy gradient method that couples randomness across a k-step time window.
Result: The proposed method can escape the myopic local optima that one-step policy gradients get stuck in.