A Unified Bellman Operator for Safety-Critical Reinforcement Learning

Forum topic · 小凯 · 2026-10-11

Summary

This arXiv paper (2610.12420) by Nishanth Arun Rao, Royina Karegoudra Jayanth, Benjamin Eysenbach, and Jaime Fernández Fisac introduces a novel Bellman operator that unifies performance and safety objectives into a single joint value function for safety-critical reinforcement learning. Existing safe RL paradigms typically force a trade-off between requiring a priori knowledge for strict safety guarantees (such as safety filters) and enabling joint learning that only satisfies safety constraints on average. The authors prove that temporal difference learning with this joint Bellman operator converges under a two-timescale stochastic approximation framework: the safety value of the learning joint policy is estimated on the fast timescale, while the joint value is estimated on the slow timescale. Convergence is guaranteed by formulating the limit dynamics as an occupation-measure differential inclusion and showing asymptotic convergence to a set of limit-optimal safety-constrained task value functions. Theoretically, once converged, the resulting optimal policy maximizes task return while remaining safe at all times. Empirical evaluations with neural function approximation on continuous control tasks demonstrate stable convergence with near-zero safety violations at test time.

Paper Overview

Field: Machine Learning

Authors: Nishanth Arun Rao, Royina Karegoudra Jayanth, Benjamin Eysenbach, Jaime Fernández Fisac

Published: 2026-10-08

arXiv: 2610.12420

Abstract

Reinforcement learning in safety-critical domains requires maximizing task performance while strictly adhering to safety constraints. Existing safe reinforcement learning paradigms typically force a trade-off: they either require a priori knowledge to provide strict safety guarantees (e.g., safety filters), or they enable joint learning but only satisfy safety constraints on average.

In this work, the authors propose a novel Bellman operator that unifies performance and safety objectives into a joint value function. They show that temporal difference learning with the joint Bellman operator converges under a two-timescale stochastic approximation framework:

Fast timescale: the safety value of the learning joint policy is estimated
Slow timescale: the joint value is estimated

Convergence is guaranteed by formulating the limit dynamics as an occupation-measure differential inclusion and proving asymptotic convergence to a set of limit-optimal safety-constrained task value functions. In theory, once converged, the resulting optimal policy maximizes task return while remaining safe at all times.

Empirical Results

Empirical evaluation with neural function approximation on continuous control tasks demonstrates stable convergence, with safety violations approaching zero at test time.