Extending MONA in Camera Dropbox: Reproduction, Learned Approval, and Reward Hacking

Forum topic · 小凯 · 2026-04-02

Summary

This arXiv paper (2603.11112) by Nathan Heath presents a reproduction-first extension of the MONA (Myopic Optimization with Non-myopic Approval) Camera Dropbox environment. MONA mitigates multi-step reward hacking by restricting the agent's planning horizon while providing far-sighted approval as a training signal. The work repackages the released codebase as a standard Python project with scripted PPO training, reproduces the published contrast between ordinary RL (91.5% reward-hacking rate) and oracle MONA (0.0% hacking rate), and introduces a modular learned-approval suite covering oracle, noisy, misspecified, learned, and calibrated approval mechanisms. In reduced-budget pilot sweeps over approval methods, horizons, dataset sizes, and calibration strategies, the best calibrated learned-supervision run achieved zero observed reward hacking, but its desired-behavior rate was far below oracle MONA (11.9% vs. 99.9%), consistent with underoptimization rather than re-emergent hacking.

Overview

Field: AI (AI safety / reward hacking) Author: Nathan Heath Published: 2026-03-31 arXiv: 2603.11112

Abstract

Myopic Optimization with Non-myopic Approval (MONA) mitigates multi-step reward hacking by restricting the agent's planning horizon while supplying far-sighted approval as a training signal. The original paper identifies a critical open question: how the method of constructing approval — particularly the degree to which approval depends on achieved outcomes — affects whether MONA's safety guarantees hold.

This paper presents a reproduction-first extension of the public MONA Camera Dropbox environment that:

1. Repackages the released codebase as a standard Python project with scripted PPO training; 2. Confirms the published contrast between ordinary RL (91.5% reward-hacking rate) and oracle MONA (0.0% hacking rate) using the released reference arrays; 3. Introduces a modular learned-approval suite covering oracle, noisy, misspecified, learned, and calibrated approval mechanisms.

In reduced-budget pilot sweeps over approval methods, horizons, dataset sizes, and calibration strategies, the best calibrated learned-supervision run achieved zero observed reward hacking, but its desired-behavior rate was far below oracle MONA (11.9% vs. 99.9%) — consistent with underoptimization rather than re-emergent hacking.

Source

arXiv: https://arxiv.org/abs/2603.11112
Auto-collected on 2026-04-02