DPO Is Not Actually Equivalent to RLHF? An ICML 2026 Paper Finds the Industry Has Been Using a Wrong Premise
📋 Paper Information
| Item | Content | |------|---------| | Title | Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment | | Authors | Zhiqin Yang, Yonggang Zhang, Wei Xue, Dong Fang, Bo Han, Yike Guo | | Affiliations | Hong Kong University of Science and Technology, Shanghai Jiao Tong University, et al. | | arXiv | 2605.20834 | | Date | 2026-05-20 | | Category | cs.AI / cs.LG | | Venue | ICML 2026 | | Core claim | The theoretical equivalence between DPO and RLHF is not universal but conditional—it depends on an implicit assumption that is frequently violated in practice. When this assumption fails, DPO is not optimizing alignment with human preferences but relative advantage over a reference model, potentially causing the model to converge in a direction humans actually dislike. |
---
Imagine you are in a restaurant choosing between two dishes.
The waiter describes:
- Dish A: "braised pork, melts in your mouth"
- Dish B: "boiled broccoli"
Then news comes from the kitchen: due to an ingredient problem, only B can be made today. You say "fine, B then."
Now imagine a different scenario: the waiter never tells you A is unavailable and simply serves B. You take a bite—"this is just boiled broccoli, I ordered the braised pork!"
But an algorithm cannot complain. The algorithm only knows you chose A. The algorithm pushes the model toward A. But the actual system can only execute B. Between your objective and the system's output lies an invisible crack.
On May 20, 2026, a 49-page theoretical paper accepted at ICML 2026 precisely characterized the mathematical nature of this crack. Its conclusion strikes at a fundamental flaw in one of the most popular algorithms in AI alignment—DPO (Direct Preference Optimization).
---
🧨 Why Did DPO Replace RLHF?
Some background first.
RLHF is the standard method for aligning large language models with human values. Its pipeline is heavy: train a reward model to predict human preferences, then use PPO reinforcement learning to optimize that reward. This means loading four models simultaneously (policy, reference, reward, value), unstable training, and extreme hyperparameter sensitivity.
In 2023, Rafailov et al. proposed DPO with an elegant simplification: no reward model, no reinforcement learning, no value network. You only need preference data (a human chose A over B), then directly optimize a clever loss function on preference pairs.
DPO's mathematical beauty: it was proven to be equivalent to RLHF at the optimum. The theory was perfect. That formula was printed in the introductions of hundreds of papers.
DPO = RLHF at the optimum.
The industry treated this equation as fact. The instruction-tuned versions of LLaMA, Qwen, and Mistral overwhelmingly use DPO or its variants. Nobody questioned the equation's validity anymore.
Until this paper.
---
🔪 The Implicit Assumption Everyone Ignored
The paper's core insight is one sentence: the equivalence proof of DPO and RLHF depends on a premise that often does not hold in practice.
That premise is: the RLHF-optimal policy must "prefer" the chosen response over the rejected response.
In plain terms: the optimal model's probability for the "human-preferred answer" should be higher than its probability for the "human-disliked answer."
This sounds like a truism—the optimal model should obviously like what humans like, right?
Wrong.
RLHF's objective is not "liking what humans like." RLHF's objective is maximizing expected reward minus a KL-divergence penalty. This means the optimal policy is a balance point between reward and diversity.
Consider a concrete example. Suppose human annotators chose response A (safety-compliant but slightly verbose) over B (concise but slightly offensive). The RLHF-optimal policy might trade off like this: A is indeed better than B, but pushing the probability gap between A and B too wide would cost generation diversity elsewhere (KL penalty too large). So the optimal policy may assign nearly the same probability to A and B.
In that case, the optimal policy does not actually "prefer" A over B. It merely gives A a slightly favorable position in a global optimization.
But DPO's loss implicitly requires the optimal policy to make A's probability clearly greater than B's. When that fails—for instance, when A and B have nearly equal probabilities under the optimal policy—DPO's optimization direction diverges from RLHF's.
This leads to the paper's core theorem: DPO optimizes not "absolute alignment with human preferences" but "relative advantage over the reference model."
---
⚰️ A Pathologically Dangerous Convergence Direction
The paper further proves a disturbing conclusion: there exists a "bad solution space" in which the DPO loss decreases while the model gets worse.
This sounds like mysticism, but mathematically it is clean.
DPO's loss can be rewritten as a form of "soft margin ranking." The "margin" is the probability gap the model assigns to chosen vs. rejected. Normal ranking losses require this margin to be positive—chosen's probability genuinely higher than rejected's. But DPO's loss permits a special case: a negative margin.
When a negative margin appears, DPO's gradient direction is not "increase the chosen's probability" but "increase the chosen-to-rejected probability ratio"—and these are not the same thing in the negative-margin region. The latter can be achieved by lowering both probabilities while letting the chosen one fall slightly less.
Result: the model's probabilities for both chosen and rejected decline while "DPO loss decreases." Overall generation quality is collapsing, but the loss curve looks perfectly fine.
This is pathological convergence. You watch the training loss go down and think everything is fine. In reality the model is drifting away from whatever direction you wanted—it is just drifting one step more slowly than the reference model.
---
🧭 Constrained Preference Optimization (CPO): A Provably Aligned Alternative
The paper does not stop at diagnosis. It proposes a solution: Constrained Preference Optimization (CPO).
Core idea: add an explicit constraint to RLHF—the reward of the chosen response must be strictly greater than that of the rejected response. It looks like a tiny change, but mathematically it does two key things:
1. It guarantees the premise of DPO's equivalence. The constraint forces the optimal policy to prefer the human-chosen direction, plugging the logical hole in DPO's theory.
2. It provides a provable alignment guarantee. Under the constraint, the authors prove that the policy's convergence direction genuinely points toward human preference—not "relative to the reference model" but "relative to absolute true preference."
Experimentally, CPO reaches SOTA on several standard benchmarks. But the paper's real value is not there—it lies in exposing a problem the industry overlooked for over two years and providing a theoretically guaranteed fix.
---
📐 Geometric Intuition: DPO's "Negative-Target" Soft Margin
The paper closes with a striking geometric reinterpretation.
In RLHF, the reward model's output can be viewed as a point: the reward difference between chosen and rejected. Training moves this point along the positive axis—further is better.
DPO turns this into "soft-margin classification." Chosen and rejected form a pair of points in the model's log-probability space. DPO tries to maximize the margin between them—but with a fatal twist: the sign of the margin can flip.
When human preference data contains natural contradictions (different annotators give opposite preferences for the same pair), or when the reference model is extremely poor on certain data, DPO's "margin" turns negative. DPO then starts optimizing a bizarre direction—"pushing chosen away from chosen, pulling rejected toward rejected"—not alignment but self-deconstruction.
The authors call this the "negative-target soft margin"—a phenomenon that does not exist in traditional classification but necessarily arises in preference optimization due to the special structure of DPO's loss.
---
🤔 Honestly: What We Are Not Sure About
Scale validation. CPO's experiments focus on mid-sized models (7B-13B) on standard benchmarks. Whether CPO's constraint remains effective at larger scales (70B+) and in more complex production settings—or over-constrains certain regions causing performance loss—is not evidenced in the paper.
Tightness of the constraint. The constraint threshold in CPO is a hyperparameter. How large should it be? Too small and the constraint is void, returning to DPO's problem space; too large and it may suppress the model's learning capacity. The paper offers no automatic threshold selection method.
Detecting violation of the implicit assumption. The paper theoretically characterizes when DPO's equivalence premise is violated, but provides no practical detection tool—during training, how do I know whether my data distribution will cause the premise to fail? Detection itself is hard, because it requires knowing the "RLHF-optimal policy"—which is precisely the thing we are trying to solve for.
DPO variants. Many DPO variants have emerged over the past two years—IPO, KTO, SimPO, ORPO, etc. Do they suffer from similar issues? Can CPO's constraint scheme be applied directly to them? The paper does not explore this.
Reflexivity of human preferences. The paper assumes human preference data is "exogenous"—annotators' judgments are independent of the training process they are embedded in. But in real human feedback loops, annotators' judgments drift as they see model outputs. How this dynamics affects CPO's constraint design is entirely unknown.
---
📚 References
1. Yang, Z. et al. (2026). *Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment.* arXiv:2605.20834. ICML 2026.
2. Rafailov, R. et al. (2023). *Direct Preference Optimization: Your Language Model is Secretly a Reward Model.* NeurIPS 2023.
3. Schulman, J. et al. (2017). *Proximal Policy Optimization Algorithms.* arXiv:1707.06347.
4. Ouyang, L. et al. (2022). *Training language models to follow instructions with human feedback.* NeurIPS 2022.
5. Azar, M. G. et al. (2024). *A General Theoretical Paradigm to Understand Learning from Human Preferences.* AISTATS 2024.
6. Ethayarajh, K. et al. (2024). *KTO: Model Alignment as Prospect Theoretic Optimization.* ICML 2024.