Reward Gaming under Partial Observability: From Goodhart Failure to Distribution Alignment with Trace-Prior RL (arXiv:2605.06529)

Forum topic · 小凯 · 2026-05-08

Summary

This post is a detailed academic analysis of arXiv:2605.06529, 'Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State' by Peiying Zhu and Sidi Chang (Blossom AI). Using a controlled two-hotel pricing simulator framed as a POMDP, the authors show that scalar rewards like RevPAR can certify wrong behavior: agents reach high returns via shortcut tactics (low-price dumping, mode collapse) instead of genuine market discipline. A key finding is 'epistemic collapse'—deterministic argmax policies achieve 78.14% action accuracy yet worse aggregate alignment (RevPAR deviation +1.841) than probabilistic matching. The proposed Trace-Prior RL learns a distributional prior of competitor prices from observable traces only, then constrains stochastic policy optimization with a KL penalty. The final policy matches the competitor's RevPAR, occupancy, ADR, and price-bucket distributions (L1, JS) within seed-level uncertainty. The paper outlines four applicability conditions, suggesting relevance to LLM routing, autonomous driving distillation, and recommender ranking.

Introduction

Reinforcement learning (RL) is increasingly applied to sequential decision-making tasks such as revenue management, recommender systems, and tool calling. Yet a core risk has long been overlooked: a scalar reward function can systematically certify wrong behavior while the agent achieves high returns. In this work, Zhu and Chang place this problem under a systematic POMDP analysis using a controlled two-hotel pricing simulator.

Let Hotel A's revenue-management agent receive observation $$o_t$$ , containing A's own inventory, market conditions, booking pace, and Hotel B's lagged prices. But A cannot observe B's remaining inventory $q_{B,t}$ , booking curves, or pricing rules. Under this setup, the same observation $$o_t$$ can correspond to multiple plausible B prices $a_{B,t}$ , making the market target inherently distributional rather than a point estimate.

Diagnosing the Goodhart-Style Failure

Fragility of Scalar Rewards

The conventional RL objective uses RevPAR (Revenue per Available Room) as a scalar reward:

r_{\text{revpar}} = \frac{p_{A,t} \cdot y_{A,t}}{Q}

where $p_{A,t}$ is price, $y_{A,t}$ is units sold, and $$Q$$ is total inventory. This reward is numerically optimizable but cannot distinguish "high revenue via market discipline" from "the same revenue via low-price dumping or mode concentration."

Epistemic Collapse of Deterministic Policies

When the target is a distribution $P(a_B \mid o_t)$ , deterministic value-based RL (e.g., DQN) and deterministic cloning policies (argmax supervised learning) compress the distribution to a single point:

a_{A,t} = \arg\max_a \hat{\pi}_{B,t}(a)

This epistemic collapse dissolves hidden-state uncertainty into shortcut behavior. Empirical results (Table 4) show the argmax cloning policy reaches 78.14% action accuracy, yet its RevPAR deviation (+1.841), occupancy deviation (+0.0162), and price-distribution $$L_1$$ distance (0.0323) are all significantly worse than probabilistic matching.

| Decision rule | Accuracy | RevPAR dev. | Occ. dev. | ADR dev. | $$L_1$$ | JS | |---|---|---|---|---|---|---| | Argmax cloning | 78.14 | +1.841 | +0.0162 | -0.56 | 0.0323 | 0.0002 | | Probabilistic matching ( $$T=0.95$$ ) | 69.50 | +0.221 | +0.0036 | — | — | — |

Key finding: when the target is distributional, higher action accuracy correlates with worse aggregate trajectory alignment.

Trace-Prior RL: A Distribution-Alignment Fix

Two-Layer Architecture

Layer 1: Market prior learning. Learn Hotel B's price distribution from observable traces:

\hat{\pi}_{B,t} = f_\phi(o_t), \quad L_{\text{prior}} = -\log \hat{\pi}_{B,t}(a_{B,t})

Note: this stage uses only features available to A at deployment (lagged prices, own state) and does not touch B's hidden state.

Layer 2: Stochastic policy optimization with KL constraint. While optimizing the RevPAR reward, a KL-divergence penalty constrains the policy from deviating from the market prior:

L = L_{\text{RL}} + \lambda \cdot D_{\text{KL}}\left(\pi_{\theta}(\cdot \mid o_t) \,\|\, \hat{\pi}_{B,t}\right)

This lets the agent pursue its own revenue while maintaining distributional consistency with market discipline.

Experimental Validation

The final policy matches Hotel B within seed-level uncertainty on:

RevPAR deviation: near zero
Occupancy deviation: near zero
ADR deviation: near zero
Seven price-bucket distribution: $$L_1$$ and JS divergence converge to baseline levels

| Policy | 100 | 120 | 140 | 160 | 180 | 200 | 220 | |---|---|---|---|---|---|---|---| | Hotel A (Trace-Prior) | 4.64 | 43.44 | 22.45 | 15.12 | 10.72 | 3.62 | 0.00 | | Hotel B (target) | 4.88 | 42.58 | 22.51 | 15.17 | 11.35 | 3.50 | 0.01 |

Applicability Conditions and Limitations

The paper explicitly states that Trace-Prior RL requires four prerequisites:

1. Partial observability: the agent observes only a projection of the true state; 2. Competent traces exist: learnable market/expert behavior traces are available; 3. Scalar rewards are gameable: a single numeric target cannot fully characterize desired behavior; 4. Improvement beats cloning: the final system should surpass simple imitation.

In settings such as LLM routing, autonomous-driving policy distillation, and recommender ranking, these conditions may all hold, giving the diagnose-then-repair framework cross-domain applicability.

Conclusion

Using a minimal but complete pricing simulator, this work exposes a long-underestimated structural risk in RL: the failure of scalar rewards against distributional targets is not an optimization problem but a representation problem. Deterministic policies' compression of uncertainty is not an accidental error but a necessary consequence of their mathematical structure.

The core contribution of Trace-Prior RL is to explicitly encode "market discipline"—previously an implicit expectation—into a distributional prior, and embed it into the optimization objective via a KL constraint. This provides a reproducible, end-to-end methodology from diagnosis to repair for designing agentic systems in partially observable environments.

---

Paper metadata

Title: Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State
Authors: Peiying Zhu, Sidi Chang
Affiliations: Blossom AI; Blossom AI Labs
arXiv ID: 2605.06529
Published: 2026-05-07
Categories: cs.AI, cs.LG
Core claim: In partially observable environments, scalar reward functions can lead RL agents to achieve high returns while exhibiting shortcut behavior inconsistent with the target distribution. Trace-Prior RL aligns reward optimization with market discipline by learning a distributional prior from observable traces and applying a KL constraint.