An Agency-Transferring Model-Free Policy Enhancement Technique for RL

Forum topic · 小凯 · 2026-06-10

Summary

This paper (arXiv:2506.04858) by Anton Bolychev, Georgiy Malaniya, and Sinan Ibrahim addresses the high cost of training reinforcement learning policies from scratch when a functional but suboptimal baseline policy already exists. The authors propose a model-free method that embeds such a baseline into the RL training process. At each step, an arbitration mechanism selects between the baseline policy and a trainable learning policy, initially relying heavily on the baseline and progressively transferring agency (decision-making authority) to the learning policy. The final result is a standalone neural network policy that requires no baseline at inference time. The paper formalizes what it means for a baseline policy to be functional: under that policy, the agent reaches a target set with high probability and stays within it. The arbitration mechanism is designed to exploit this property, achieving high target-reaching rates from early training. Theoretical analysis provides a formal justification and derives an explicit lower bound on the target-reaching probability of the final standalone policy. Experiments on continuous control benchmarks show returns comparable to or better than competing methods while maintaining the highest target-reaching rate throughout training.

An Agency-Transferring Model-Free Policy Enhancement Technique

Field: Machine Learning Authors: Anton Bolychev, Georgiy Malaniya, Sinan Ibrahim Published: 2025-06-06 arXiv: 2506.04858

Abstract

Training reinforcement learning (RL) policies from scratch is costly: it requires careful reward and environment design, extensive tuning, and substantial computation. Yet many control problems already have a functional but suboptimal policy available as a baseline. This paper proposes a method for embedding such a baseline into the RL training process, simultaneously improving training efficiency relative to from-scratch methods and producing a learning policy that outperforms the baseline.

At each step, the method arbitrates between the baseline policy and a trainable learning policy, initially relying strongly on the baseline policy and then progressively transferring agency to the learning policy. By the end of training, the learning policy is a standalone neural network that operates without the baseline policy.

Formalizing Baseline Functionality

The paper formalizes what it means for a baseline policy to be functional: under such a policy, the agent reaches a target set with high probability and remains within it. The proposed arbitration mechanism is designed to exploit this property, achieving high target-reaching rates from the very beginning of training.

Theoretical Guarantees

The theoretical analysis formally explains this behavior under the stated assumptions and extends it to the final baseline-free phase, deriving an explicit lower bound on the target-reaching probability of the standalone learning policy.

Experimental Results

Experiments on continuous control benchmarks demonstrate that the method achieves returns comparable to or higher than competing methods, while maintaining the highest target-reaching rate throughout the entire training process.

--- *Auto-collected on 2026-06-10*