Summary
This paper introduces Redistribution-based Cost Inference (RCI), a framework for safe offline reinforcement learning when only sparse, trajectory-level supervision is available. Instead of dense per-step cost annotations, the supervisor provides a single binary stop signal at the first unsafe transition, with no per-step attribution. The authors frame this as a temporal credit assignment problem and use return decomposition to redistribute the sparse stop feedback into dense per-step costs, then train a constrained offline policy on the augmented dataset. Theoretically, they prove that return-equivalent redistributions preserve both the set of feasible policies in a constrained MDP and the optimal Lagrangian, establishing that the transformation is loss-free. Practically, the densified costs yield better-conditioned cost critic learning. Experiments on highway driving and robot manipulation show significantly reduced violation rates compared to sparse and classifier-based baselines, with robustness to heterogeneous dataset composition and label noise. The work is by Ebenezer Gelo, Geraud Nangue Tasse, and Steven James, available as arXiv:2508.03417.
Paper Overview
Field: Machine Learning
Authors: Ebenezer Gelo, Geraud Nangue Tasse, Steven James
Published: 2026-08-13
arXiv: 2508.03417
Summary
Safe offline RL typically assumes access to dense per-step cost annotations, but in practice supervisors only provide trajectory-level stop feedback: a single binary signal at the first unsafe transition, with no per-step attribution. The authors frame this as a temporal credit assignment problem and propose the Redistribution-based Cost Inference (RCI) framework, which uses return decomposition to convert the sparse stop feedback into dense per-step costs, and then trains a constrained offline policy on the augmented dataset.
Theoretical Contribution
The authors prove that return-equivalent redistributions preserve both the set of feasible policies in a constrained MDP (CMDP) and the optimal Lagrangian. This establishes that the proposed transformation is theoretically loss-free, while in practice producing better-conditioned cost critic learning.
Experimental Results
- Evaluated on highway driving and robot manipulation tasks
- Compared against sparse and classifier-based baselines
- Shows significantly reduced violation rates
- Demonstrates robustness to heterogeneous dataset composition and label noise
Links
- Paper: https://arxiv.org/abs/2508.03417
---
*Auto-collected on 2026-08-14*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178633455