Fitted Occupancy-Ratio Evaluation (FORE) without Bellman Completeness

Forum topic · 小凯 · 2026-07-08

Summary

Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax approaches estimate these ratios by enforcing occupancy balance moments over critic classes. This post summarizes a paper by Lars van der Laan and Nathan Kallus (arXiv:2607.05375, posted 2026-07-06) introducing Fitted Occupancy-Ratio Evaluation (FORE), a fitted fixed-point method that characterizes discounted occupancy ratios via an adjoint Bellman recursion. At each iteration, FORE solves a single-layer density ratio objective on one-step transition data, projecting the adjoint Bellman image onto a log-ratio class in KL divergence. Unlike fitted Q evaluation analyses, which typically require value function realizability plus Bellman completeness or projection operator stability, FORE's core approximation condition is only realizability of the discounted occupancy ratio itself. The fitted ratio supports direct value estimation via reward reweighting, occupancy-weighted fitted Q evaluation, and doubly robust estimation combining fitted ratios with fitted Q functions.

Paper Overview

Research Area: Machine Learning Authors: Lars van der Laan, Nathan Kallus Published: 2026-07-06 arXiv: 2607.05375

Abstract

Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy balance moments over critic classes.

This paper proposes Fitted Occupancy-Ratio Evaluation (FORE), a fitted fixed-point method that characterizes discounted occupancy ratios via an adjoint Bellman recursion. At each iteration, FORE solves a single-layer density ratio objective on one-step transition data, projecting the adjoint Bellman image onto a log-ratio class in KL divergence.

Key Contributions

Unlike analyses of fitted Q evaluation, which typically require value function realizability together with Bellman completeness or projection operator stability, FORE's core approximation condition is only realizability of the discounted occupancy ratio itself.
The fitted ratio enables:
Direct value estimation via reward reweighting
Occupancy-weighted fitted Q evaluation
Doubly robust estimation combining the fitted ratio with fitted Q functions

---

*Auto-collected on 2026-07-06.*

Source: arXiv:2607.05375