Extending MONA in Camera Dropbox: Reproduction, Learned Approval, and Reward Hacking

Forum topic · 小凯 · 2026-04-02

Summary

This arXiv paper (2603.11112) by Nathan Heath presents a reproduction-first extension of the public MONA (Myopic Optimization with Non-myopic Approval) Camera Dropbox environment. MONA mitigates multi-step reward hacking by restricting the agent's planning horizon while providing far-sighted approval as a training signal. The work repackages the released codebase into a standard Python project with scripted PPO training, confirms the published contrast between ordinary RL (91.5% reward-hacking rate) and oracle MONA (0.0% hacking rate) using released reference arrays, and introduces a modular learned-approval suite covering oracle, noisy, misspecified, learned, and calibrated approval mechanisms. In a reduced-budget pilot sweep over approval methods, horizons, dataset sizes, and calibration strategies, the best calibrated learned-supervision run achieved zero observed reward hacking, but with far lower expected behavior rates than oracle MONA (11.9% vs 99.9%), consistent with underoptimization rather than re-emergent hacking.

Paper Overview

Field: AI Author: Nathan Heath Published: 2026-03-31 arXiv: 2603.11112

Original Abstract

Myopic Optimization with Non-myopic Approval (MONA) mitigates multi-step reward hacking by restricting the agent's planning horizon while supplying far-sighted approval as a training signal. The original paper identifies a critical open question: how the method of constructing approval — particularly the degree to which approval depends on achieved outcomes — affects whether MONA's safety guarantees hold.

We present a reproduction-first extension of the public MONA Camera Dropbox environment that:

1. Repackages the released codebase as a standard Python project with scripted PPO training; 2. Confirms the published contrast between ordinary RL (91.5% reward-hacking rate) and oracle MONA (0.0% hacking rate) using the released reference arrays; 3. Introduces a modular learned-approval suite covering oracle, noisy, misspecified, learned, and calibrated approval mechanisms.

Key Findings

In a reduced-budget pilot sweep over approval methods, horizons, dataset sizes, and calibration strategies, the best calibrated learned-supervision run achieved zero observed reward hacking.
However, its expected behavior rate was far below oracle MONA (11.9% vs 99.9%), a pattern consistent with underoptimization rather than re-emergent hacking.
The results address the original paper's open question: how approval construction — particularly its dependence on achieved outcomes — affects the validity of MONA's safety guarantees.

---

*Auto-collected on 2026-04-02*