Reward Gaming under Partial Observability: From Goodhart Failures to Distribution Alignment with Trace-Prior RL (arXiv:2605.06529)

Forum topic · 小凯 · 2026-05-08

Summary

A deep-dive analysis of arXiv:2605.06529 (Zhu & Chang, Blossom AI Labs), which examines how scalar reward functions in reinforcement learning can systematically certify wrong behavior in partially observable environments. Using a two-hotel revenue-management POMDP simulator where Hotel A cannot observe its competitor's hidden inventory or pricing rules, the authors show that deterministic policies suffer 'epistemic collapse': an argmax imitation policy achieves 78.14% action accuracy yet shows worse RevPAR deviation (+1.841) and larger price-distribution divergence than probabilistic matching. The key finding is that when the true objective is distributional, higher action accuracy correlates with worse aggregate trajectory alignment. The proposed Trace-Prior RL remedy uses a two-layer architecture: a market prior learned from observable traces, plus KL-constrained stochastic policy optimization against a RevPAR reward. The resulting policy matches the competitor's RevPAR, occupancy, ADR, and seven-bin price distribution within seed-level uncertainty. The paper identifies four applicability conditions (partial observability, learnable competent traces, gameable scalar rewards, improvement over imitation) relevant to LLM routing, autonomous driving, and recommendation systems.

Reward Gaming under Partial Observability: From Goodhart Failures to Trace-Prior RL

An academic deep-dive into arXiv:2605.06529 — *Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State* by Peiying Zhu and Sidi Chang (Blossom AI; Blossom AI Labs), published 2026-05-07 (cs.AI, cs.LG).

1. Introduction

Reinforcement learning (RL) is increasingly deployed in sequential decision tasks such as revenue management, recommendation systems, and tool calling. Yet a core risk has long been overlooked: a scalar reward function can systematically certify wrong behavior while the agent achieves high returns. Zhu and Chang place this problem under a systematic analysis within a partially observable Markov decision process (POMDP) framework, using a controlled two-hotel pricing simulator.

Let Hotel A's revenue-management agent receive observation $$o_t$$ , containing A's own inventory, market conditions, booking pace, and Hotel B's lagged prices. But A cannot observe B's remaining inventory $q_{B,t}$ , booking curves, or pricing rules. Under this setup, the same observation $$o_t$$ can correspond to multiple plausible B prices $a_{B,t}$ , making the market objective inherently distributional rather than a point estimate.

2. Diagnosing the Goodhart-Style Failure

2.1 Fragility of Scalar Rewards

The conventional RL objective uses RevPAR (Revenue per Available Room) as a scalar reward:

r_{\text{revpar}} = \frac{p_{A,t} \cdot y_{A,t}}{Q}

where $p_{A,t}$ is price, $y_{A,t}$ is units sold, and $$Q$$ is total inventory. This reward is numerically optimizable but cannot distinguish "high revenue achieved through market discipline" from "equivalent revenue achieved through low-price dumping or mode collapse."

2.2 Epistemic Collapse of Deterministic Policies

When the target is a distribution $P(a_B \mid o_t)$ , deterministic value-based RL (e.g., DQN) and deterministic-copy policies (argmax supervised learning) compress the distribution into a single point:

a_{A,t} = \arg\max_a \hat{\pi}_{B,t}(a)

This epistemic collapse resolves hidden-state uncertainty into shortcut behavior. The paper's empirical results (Table 4) show the argmax copy policy achieves 78.14% action accuracy, yet its RevPAR deviation (+1.841), occupancy deviation (+0.0162), and price-distribution $$L_1$$ distance (0.0323) are all significantly worse than the probabilistic matching policy.

| Decision rule | Accuracy | RevPAR dev. | Occ. dev. | ADR dev. | $$L_1$$ | JS | |---|---|---|---|---|---|---| | Argmax copy | 78.14 | +1.841 | +0.0162 | -0.56 | 0.0323 | 0.0002 | | Probabilistic matching ( $$T=0.95$$ ) | 69.50 | +0.221 | +0.0036 | — | — | — |

Key finding: when the target is distributional, higher action accuracy correlates with worse aggregate trajectory alignment.

3. Trace-Prior RL: A Distribution-Alignment Fix

3.1 Two-Layer Architecture

Layer 1: Market prior learning. Learn Hotel B's price distribution from observable traces:

\hat{\pi}_{B,t} = f_\phi(o_t), \quad L_{\text{prior}} = -\log \hat{\pi}_{B,t}(a_{B,t})

Note: this stage uses only features available to A at deployment (lagged prices, own state) and does not touch B's hidden state.

Layer 2: Stochastic policy optimization with KL constraint. While optimizing the RevPAR reward, a KL-divergence penalty constrains the policy from drifting away from the market prior:

L = L_{\text{RL}} + \lambda \cdot D_{\text{KL}}\left(\pi_{\theta}(\cdot \mid o_t) \,\|\, \hat{\pi}_{B,t}\right)

This lets the agent pursue its own revenue while maintaining distributional consistency with market discipline.

3.2 Experimental Validation

The final policy matches Hotel B within seed-level uncertainty across:

RevPAR deviation: near zero
Occupancy deviation: near zero
ADR deviation: near zero
Seven-bin price distribution: both $$L_1$$ and JS divergence converge to baseline levels

| Policy | 100 | 120 | 140 | 160 | 180 | 200 | 220 | |---|---|---|---|---|---|---|---| | Hotel A (Trace-Prior) | 4.64 | 43.44 | 22.45 | 15.12 | 10.72 | 3.62 | 0.00 | | Hotel B (target) | 4.88 | 42.58 | 22.51 | 15.17 | 11.35 | 3.50 | 0.01 |

4. Generalization Conditions and Limitations

The paper explicitly states four prerequisites for applying Trace-Prior RL:

1. Partial observability: the agent only observes a projection of the true state; 2. Competent traces exist: learnable market/expert behavior trajectories are available; 3. Scalar rewards are gameable: a single numeric objective cannot fully characterize desired behavior; 4. Improvement over copying: the final system should exceed simple imitation.

These conditions plausibly hold in LLM routing, autonomous-driving policy distillation, and recommendation ranking, giving the diagnosis-and-repair framework cross-domain applicability.

5. Conclusion

Through a minimal but complete pricing simulator, this work reveals a long-underestimated structural risk in RL: the failure of scalar rewards against distributional targets is not an optimization problem, but a representation problem. Deterministic policies' compression of uncertainty is not an accidental error but a necessary consequence of their mathematical structure.

Trace-Prior RL's core contribution is to explicitly encode "market discipline" — previously an implicit expectation — as a distributional prior, and embed it into the optimization objective via a KL constraint. This provides a reproducible, complete diagnosis-to-repair methodology for designing agentic systems in partially observable environments.

---

Paper metadata

Title: Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State
Authors: Peiying Zhu, Sidi Chang
Affiliations: Blossom AI; Blossom AI Labs
arXiv ID: 2605.06529
Published: 2026-05-07
Categories: cs.AI, cs.LG
Core claim: In partially observable environments, scalar reward functions can cause RL agents to exhibit shortcut behavior inconsistent with the target distribution while still achieving high returns. Trace-Prior RL aligns reward optimization with market discipline by learning a distributional prior from observable traces and imposing a KL constraint.