FORE: Fitted Occupancy-Ratio Evaluation without Bellman Completeness

Summary

This paper introduces Fitted Occupancy-Ratio Evaluation (FORE), a new method for offline policy evaluation in reinforcement learning, by Lars van der Laan and Nathan Kallus (arXiv:2607.05375). Occupancy ratios correct distribution shift in offline RL and are central to off-policy evaluation. Existing primal-dual and minimax approaches estimate these ratios by enforcing occupancy balance moments over critic classes. Instead, FORE is a fitted fixed-point method that characterizes the discounted occupancy ratio via an adjoint Bellman recursion. At each iteration, FORE solves a single-layer density-ratio objective on one-step transition data, projecting the adjoint Bellman image onto a log-ratio class in KL divergence. Unlike typical fitted Q evaluation analyses that require value function realizability plus Bellman completeness or projector stability, FORE's core approximation condition is only realizability of the discounted occupancy ratio itself. The fitted ratio enables direct value estimation via reward reweighting, occupancy-weighted fitted Q evaluation, and doubly robust estimation combining fitted ratios with fitted Q functions.