Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning

Forum topic · 小凯 · 2026-08-27

Summary

This arXiv paper (2608.24858) by Lars van der Laan and Nathan Kallus introduces isotonic Bellman calibration for marginalized importance weighting in offline reinforcement learning. Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, which is characterized by an adjoint Bellman equation. Existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations caused by function-class approximation, regularization, or incomplete optimization. These violations are hard to diagnose and reduce because the objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping. The proposed method is a one-dimensional, model-agnostic post-processing step that reduces occupancy-balance violations while preserving the ranking information in any initial occupancy-ratio estimate. It works by applying fitted occupancy ratio evaluation (FORE) over a class of one-dimensional non-decreasing transformations to correct the scale and shape of the estimates. The authors characterize Bellman calibration as a conditional fixed-point property equivalent to occupancy balance of the calibrated ratio for every test function.

Paper Overview

Field: Machine Learning Authors: Lars van der Laan, Nathan Kallus Published: 2026-08-25 arXiv: 2608.24858

Background

Marginalized importance weighting evaluates a target policy in offline reinforcement learning by reweighting offline state-action samples with the policy's discounted occupancy ratio. This occupancy ratio is characterized by an adjoint Bellman equation.

Problem

Existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations due to:

Function-class approximation
Regularization
Incomplete optimization

These violations are difficult to diagnose and reduce because the objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping.

Proposed Method

The authors introduce isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method that:

Reduces occupancy-balance violations while preserving the ranking information in any initial occupancy-ratio estimate
Applies fitted occupancy ratio evaluation (FORE) over a class of one-dimensional non-decreasing transformations to correct the estimated ratio's scale and shape

Theoretical Characterization

Bellman calibration is characterized as a conditional fixed-point property, equivalent to occupancy balance of the calibrated ratio for every test function.

Abstract (Original)

Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characterized by an adjoint Bellman equation. Existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations because of function-class approximation, regularization, or incomplete optimization. These violations are difficult to diagnose and reduce because the objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping. We introduce isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method that reduces these violations while preserving the ranking information in any initial occupancy-ratio estimate. The method applies fitted occupancy ratio evaluation (FORE) over a class of one-dimensional non-decreasing transformations to correct the scale and shape of the estimate.

--- *Auto-collected on 2026-08-27*