Redistribution-based Cost Inference for Sparse Safe Offline Reinforcement Learning

Forum topic · 小凯 · 2026-08-14

Summary

This paper addresses safe offline reinforcement learning under sparse trajectory-level supervision, where supervisors provide only a binary signal at the first unsafe transition instead of dense per-step cost labels. The authors frame this as a temporal credit assignment problem and propose Redistribution-based Cost Inference (RCI), a framework that uses return decomposition to convert sparse stopping feedback into dense per-step costs, then trains constrained offline policies on the augmented dataset. The authors prove that return-equivalent redistribution preserves the feasible policy set and optimal Lagrangian in CMDPs, establishing that the transformation is lossless in theory while yielding better-conditioned cost critic learning in practice. Experiments on highway driving and robotic manipulation tasks show substantially lower violation rates compared to sparse and classifier-based baselines, with robustness to heterogeneous dataset composition and label noise. The work was published on arXiv in August 2026.

Paper Overview

Research Area: Machine Learning (ML) Authors: Ebenezer Gelo, Geraud Nangue Tasse, Steven James Release Date: 2026-08-13 arXiv: 2508.03417

Abstract

Safe offline reinforcement learning typically assumes access to dense per-step cost annotations. In practice, however, supervisors often provide only trajectory-level stopping feedback: a binary signal at the first unsafe transition, without per-step attribution. The authors frame this problem as a temporal credit assignment problem and propose a Redistribution-based Cost Inference (RCI) framework.

The approach uses return decomposition to convert sparse stopping feedback into dense per-step costs, and then trains constrained offline policies on the augmented dataset.

Theoretical Contribution

The authors prove that return-equivalent redistribution preserves:

The feasible policy set in constrained Markov decision processes (CMDPs)
The optimal Lagrangian value

This establishes that the transformation is lossless in theory, while in practice producing better-conditioned cost critic learning.

Empirical Results

Experiments on:

Highway driving
Robotic manipulation

show that, compared to sparse and classifier-based baselines, the proposed method achieves:

Significantly lower violation rates
Robustness to heterogeneous dataset composition
Robustness to label noise

Key Points

Problem Setting: Safe offline RL with trajectory-level binary safety feedback rather than dense per-step costs.
Method: RCI redistributes a single terminal unsafe signal across preceding steps using return decomposition.
Theory: Return-equivalent redistribution preserves CMDP feasible policies and optimal Lagrangian, guaranteeing no information loss.
Practice: Cost critic training becomes better-conditioned on the redistributed dense labels.
Evaluation: Tested on highway driving and robotic manipulation benchmarks.
Outcomes: Lower violation rates than sparse and classifier-based baselines; robust to dataset heterogeneity and label noise.

--- *Auto-collected on 2026-08-14*

#paper #arXiv #ML