Summary
This paper introduces isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method for marginalized importance weighting in offline reinforcement learning. Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, which is characterized by an adjoint Bellman equation. However, existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations caused by function-class approximation, regularization, or incomplete optimization. These violations are hard to diagnose and reduce because such objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping. The proposed calibration method reduces these violations while preserving the ranking information in any initial occupancy-ratio estimate, applying fitted occupancy-ratio evaluation (FORE) over the class of one-dimensional non-decreasing transformations to correct the scale and shape of estimates. The authors characterize Bellman calibration as a conditional fixed-point property equivalent to occupancy balance of the calibrated ratio for every test function. Work by Lars van der Laan and Nathan Kallus, available as arXiv:2608.24858.
Paper Overview
Field: Machine Learning
Authors: Lars van der Laan, Nathan Kallus
Published: 2026-08-25
arXiv: 2608.24858
Summary
Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characterized by an adjoint Bellman equation. Existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations because of function-class approximation, regularization, or incomplete optimization. These violations are difficult to diagnose and reduce because the objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping.
Contribution
The authors introduce isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method that reduces these violations while preserving the ranking information in any initial occupancy-ratio estimate. The method corrects the estimated scale and shape by applying fitted occupancy-ratio evaluation (FORE) to the class of one-dimensional non-decreasing transformations.
Theoretical Result
Bellman calibration is characterized as a conditional fixed-point property: it is equivalent to the occupancy balance of the calibrated ratio for every test function.
---
*Source: zhichai.net forum post, auto-collected 2026-08-27.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178634089