English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Redistribution-based Cost Inference Improves Sparse Safe Offline RL

Forum topic · 小凯 · 2026-08-14

Summary

This paper introduces Redistribution-based Cost Inference (RCI), a framework for safe offline reinforcement learning when only sparse, trajectory-level supervision is available. Instead of dense per-step cost annotations, the supervisor provides a single binary stop signal at the first unsafe transition, with no per-step attribution. The authors frame this as a temporal credit assignment problem and use return decomposition to redistribute the sparse stop feedback into dense per-step costs, then train a constrained offline policy on the augmented dataset. Theoretically, they prove that return-equivalent redistributions preserve both the set of feasible policies in a constrained MDP and the optimal Lagrangian, establishing that the transformation is loss-free. Practically, the densified costs yield better-conditioned cost critic learning. Experiments on highway driving and robot manipulation show significantly reduced violation rates compared to sparse and classifier-based baselines, with robustness to heterogeneous dataset composition and label noise. The work is by Ebenezer Gelo, Geraud Nangue Tasse, and Steven James, available as arXiv:2508.03417.

Paper Overview

Field: Machine Learning Authors: Ebenezer Gelo, Geraud Nangue Tasse, Steven James Published: 2026-08-13 arXiv: 2508.03417

Summary

Safe offline RL typically assumes access to dense per-step cost annotations, but in practice supervisors only provide trajectory-level stop feedback: a single binary signal at the first unsafe transition, with no per-step attribution. The authors frame this as a temporal credit assignment problem and propose the Redistribution-based Cost Inference (RCI) framework, which uses return decomposition to convert the sparse stop feedback into dense per-step costs, and then trains a constrained offline policy on the augmented dataset.

Theoretical Contribution

The authors prove that return-equivalent redistributions preserve both the set of feasible policies in a constrained MDP (CMDP) and the optimal Lagrangian. This establishes that the proposed transformation is theoretically loss-free, while in practice producing better-conditioned cost critic learning.

Experimental Results

  • Evaluated on highway driving and robot manipulation tasks
  • Compared against sparse and classifier-based baselines
  • Shows significantly reduced violation rates
  • Demonstrates robustness to heterogeneous dataset composition and label noise
  • Links

  • Paper: https://arxiv.org/abs/2508.03417
--- *Auto-collected on 2026-08-14*

Tags

#reinforcement-learning#offline-rl#safe-rl#credit-assignment#cost-inference#constrained-mdp#machine-learning#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633455