Paper: Fitted Occupancy-Ratio Evaluation without Bellman Completeness (FORE)

Summary

Fitted Occupancy-Ratio Evaluation (FORE) is a new method by Lars van der Laan and Nathan Kallus for estimating occupancy ratios in offline reinforcement learning, addressing distribution shift in off-policy evaluation (arXiv:2607.05375). Unlike existing primal-dual and minimax approaches that enforce occupancy-balance moments over critic classes, FORE is a fitted fixed-point method that characterizes the discounted occupancy ratio via an adjoint Bellman recursion. At each iteration it solves a single-layer density-ratio objective on one-step transition data, projecting the adjoint Bellman image onto a log-ratio class in KL divergence. Crucially, whereas analyses of fitted Q evaluation typically require value-function realizability plus Bellman completeness or projector stability, FORE's core approximation condition is only realizability of the discounted occupancy ratio itself. The fitted ratio enables direct value estimation via reward reweighting, occupancy-weighted fitted Q evaluation, and doubly robust estimators combining fitted ratios with fitted Q functions.