Paper Information
| Item | Content | |------|------| | Title | Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment | | Authors | Zhiqin Yang, Yonggang Zhang, Wei Xue, Dong Fang, Bo Han, Yike Guo | | Institutions | Hong Kong University of Science and Technology, Shanghai Jiao Tong University, et al. | | arXiv | 2605.20834 | | Date | 2026-05-20 | | Categories | cs.AI / cs.LG | | Venue | ICML 2026 | | Core claim | The theoretical equivalence of DPO and RLHF is not universal but conditional—it depends on an implicit assumption that is often violated in practice. When it fails, DPO is not optimizing alignment with human preferences but relative advantage over the reference model, potentially causing the model to converge in a direction humans actually dislike. |
---
A Restaurant Analogy
Imagine you're at a restaurant choosing between two dishes. The waiter describes:
- Dish A: "braised pork, melts in your mouth"
- Dish B: "boiled broccoli"
Then a message comes from the kitchen: due to an ingredient shortage, only B can be made today. You say "fine, B then."
Now imagine a different scenario: the waiter never tells you A is unavailable and simply serves B. You take a bite—"this is just boiled broccoli! I wanted braised pork!"
But algorithms don't complain. The algorithm only knows you chose A, so it pushes you toward A. Yet the deployed system can only execute B. There's an invisible crack between the objective you're optimizing and the output you actually get.
On May 20, 2026, a 49-page theoretical paper accepted at ICML 2026 precisely characterized the mathematical nature of this crack. Its conclusion strikes at a fundamental flaw in one of AI alignment's most popular algorithms: DPO.
---
Why Did DPO Replace RLHF?
Some background first.
RLHF is the standard method for aligning large language models with human values. Its pipeline is heavy: train a reward model to predict human preferences, then use PPO reinforcement learning to optimize that reward. This means loading four models simultaneously (policy, reference, reward, value), unstable training, and extreme hyperparameter sensitivity.
In 2023, Rafailov et al. proposed DPO with an elegant simplification: no reward model, no reinforcement learning, no value network. You only need preference data (humans chose A over B) and directly optimize a clever loss on preference pairs.
DPO's mathematical beauty: it was proven equivalent to RLHF at the optimum. The theory was perfect. That formula appeared in the introductions of hundreds of papers.
DPO = RLHF at the optimum.
The industry treated this equation as fact. The instruction-tuned versions of LLaMA, Qwen, and Mistral mostly use DPO or its variants. Nobody questioned the equation's validity.
Until this paper.
---
The Implicit Assumption Everyone Ignored
The paper's core insight is one sentence: the equivalence proof between DPO and RLHF relies on a premise that often does not hold in practice.
That premise is: the RLHF-optimal policy must prefer the chosen response over the rejected response.
In plain terms: the optimal model should assign higher probability to the human-preferred answer than to the human-dispreferred one.
This sounds trivially true—the optimal model obviously prefers what humans prefer, right?
Wrong.
RLHF's objective is not "like what humans like." RLHF maximizes expected reward minus a KL-divergence penalty. This means the optimal policy is a balance point between reward and diversity.
Consider a concrete example. Suppose an annotator chose response A (safe-compliant but slightly verbose) over B (concise but slightly offensive). The RLHF-optimal policy might trade off as follows: A is indeed better than B, but pushing the probability gap between A and B too wide would hurt generation diversity elsewhere (KL penalty too large). So the optimal policy may assign nearly equal probabilities to A and B.
In that case, the optimal policy doesn't actually "prefer" A over B. It merely gives A a slightly favorable position in the global optimization.
But DPO's loss implicitly requires the optimal policy to give A clearly higher probability than B. When that requirement fails—e.g., near-equal probabilities at the optimum—DPO's optimization direction diverges from RLHF.
Hence the paper's core theorem: DPO optimizes not absolute alignment with human preferences, but relative advantage over the reference model.
---
A Pathologically Dangerous Convergence Direction
The paper proves an unsettling result: there exists a 'bad solution space' in which the DPO loss decreases while the model gets worse.
It sounds mystical, but the math is clean.
DPO's loss can be rewritten as a form of "soft margin ranking." The "margin" is the probability gap between chosen and rejected. Normal ranking losses require a positive margin—chosen probability genuinely higher. But DPO's loss permits a special case: negative margin.
When the negative-margin region appears, DPO's gradient direction is not "increase the chosen response's probability" but "increase the chosen-to-rejected probability ratio"—and these are not the same thing in the negative-margin region. The latter can be achieved by decreasing both probabilities while letting the chosen one decrease slightly less.
Result: the DPO loss drops while the model's probabilities for both chosen and rejected responses decline. Overall generation quality collapses, yet the loss curve looks fine.
This is "pathological convergence." Training loss goes down, everything seems fine—but the model is drifting away from anything you wanted. It's just drifting one step slower than the reference model.
---
Constrained Preference Optimization (CPO): A Provably Aligned Alternative
The paper doesn't stop at diagnosis. It proposes Constrained Preference Optimization (CPO).
Core idea: add an explicit constraint to RLHF—the chosen response's reward must be strictly greater than the rejected response's reward. It looks like a minor tweak, but mathematically it does two critical things:
1. It guarantees DPO's equivalence premise holds. The constraint forces the optimal policy to prefer the human-chosen direction, closing the logical gap in DPO's theory.
2. It provides a provable alignment guarantee. Under the constraint, the authors prove the policy's convergence direction genuinely points toward human preference—not relative to the reference model, but relative to the true preference.
Experimentally, CPO achieves SOTA on multiple standard benchmarks. But the paper's real value is not there—it lies in exposing a problem the industry ignored for over two years and providing a theoretically guaranteed fix.
---
Geometric Intuition: DPO's "Negative-Target" Soft Margin
The paper closes with an elegant geometric reinterpretation.
In RLHF, the reward model's output can be seen as a point: the reward difference between chosen and rejected. Training moves this point along the positive half-axis—farther is better.
DPO transforms this into "soft-margin classification." Chosen and rejected form a pair of points in the model's log-probability space. DPO maximizes the margin between them—but with a fatal distortion: the margin's sign can flip.
When human preference data contains natural contradictions (different annotators giving opposite preferences on the same pair), or when the reference model is extremely poor on certain data, DPO's margin turns negative. DPO then optimizes a strange direction of "pushing chosen away from chosen, pulling rejected toward rejected"—not alignment, but self-deconstruction.
The authors call this the "negative-target soft margin"—a phenomenon that doesn't exist in classical classification but necessarily arises in preference optimization due to DPO's loss structure.
---
Honestly: What We're Not Sure About
Scale validation. CPO's experiments cover 7B–13B models on standard benchmarks. Whether the constraint remains effective at larger scale (70B+) and in complex production settings—or over-constrains and hurts performance—is not addressed.
Constraint tightness. The constraint threshold is a hyperparameter. Too small and the constraint is ineffective, returning to DPO's problem space; too large and it may suppress learning. The paper offers no automatic threshold selection method.
Detecting assumption violations. The paper theoretically characterizes when DPO's equivalence premise fails but provides no practical detection tool—during training, how do I know whether my data distribution will break the premise? That detection is hard in itself, since it requires knowing the RLHF-optimal policy—which is precisely the problem we're trying to solve.
DPO variants. Many variants have emerged—IPO, KTO, SimPO, ORPO, etc. Do they share similar problems? Can CPO's constraint be applied directly to them? The paper doesn't explore this.
Reflexivity of human preferences. The paper assumes preference data is "exogenous"—annotator judgments are independent of the training process they're embedded in. In real feedback loops, annotators' judgments drift as they see model outputs. How this dynamics affects CPO's constraint design is entirely unknown.
---
References
1. Yang, Z. et al. (2026). *Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment.* arXiv:2605.20834. ICML 2026. 2. Rafailov, R. et al. (2023). *Direct Preference Optimization: Your Language Model is Secretly a Reward Model.* NeurIPS 2023. 3. Schulman, J. et al. (2017). *Proximal Policy Optimization Algorithms.* arXiv:1707.06347. 4. Ouyang, L. et al. (2022). *Training language models to follow instructions with human feedback.* NeurIPS 2022. 5. Azar, M. G. et al. (2024). *A General Theoretical Paradigm to Understand Learning from Human Preferences.* AISTATS 2024. 6. Ethayarajh, K. et al. (2024). *KTO: Model Alignment as Prospect Theoretic Optimization.* ICML 2024.