Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking in Online Adaptation
Research area: Alignment Authors: Subramanyam Sahoo, Aman Chadha, Vinija Jain arXiv: 2507.00007
Abstract
Conservative offline training is widely advocated as a safe foundation for subsequent online adaptation: if a policy stays close to well-supported behaviour, the argument goes, it is less likely to exploit imperfections in a learned reward model. The authors challenge this intuition empirically and mechanistically.
They train a Qwen3-14B policy under Direct Preference Optimisation (DPO) with three levels of conservatism (β ∈ {β_lo, β_mid, β_hi} derived from empirical log-ratio percentiles), then adapt each checkpoint online against a learned reward ensemble (3× Qwen3-1.7B) while measuring true performance on GSM8K exact-answer accuracy.
Key findings
- Higher offline conservatism monotonically increases reward-hacking damage, measured by the Goodhart gap and its area under the curve (AUGC), with Spearman ρ = 1.0 across all three conditions.
- A three-link causal mechanism explains the effect: 1. High-β DPO compresses policy entropy. 2. Low-entropy policies generate less diverse responses, concentrated in a narrow region of the reward model's training distribution (lower pairwise cosine distance). 3. Despite this proximity, ensemble disagreement (epistemic uncertainty) increases with β and is exploited faster during online optimisation.
- Power-law fit: The authors fit power-law curves to the (β, AUGC) data and identify a practical optimal conservatism level β* that balances alignment fidelity against manipulation vulnerability.
Conclusion
The field should aim for calibrated conservatism rather than maximised conservatism when preparing policies for online RL adaptation.
--- *Auto-collected on 2026-07-01*