Paper Overview
- Field: Machine Learning
- Authors: Ruoming Jin, Xinyu Li, Hao Zhou, Jianfeng Zhu, Ruixin Guo, Feodor Dragan, Lei Xu, Haixun Wang, Yang Zhou
- Published: 2026-10-05
- arXiv: 2610.00061
- Experiments on personalized text generation benchmarks show GAP-DPO consistently outperforms standard DPO variants in style fidelity, preference alignment, and generation quality.
- The results establish gradient alignment as a unifying principle for personalized preference optimization.
- Preference pair selection should be treated as an intrinsic component of the optimization geometry, not a heuristic preprocessing step.
Summary
Personalizing large language models (LLMs) requires aligning generation behavior with user-specific preferences rather than aggregate quality. While Direct Preference Optimization (DPO) provides a stable framework for preference learning, its effectiveness in personalized settings critically depends on how preference pairs are selected. Existing approaches typically rely on heuristic criteria, such as likelihood-based extremes, which decouple optimization from explicit user utility and can lead to degraded personalization.
The authors formalize personalized preference learning as a geometry-aligned optimization problem by analyzing the first-order interaction between gradients of expected user utility and DPO update directions. Their analysis reveals that, under off-policy sampling, when the preference margin aligns with the utility gradient direction, the DPO update transitions from a pure error-correction signal to a reinforcement-like update.
This perspective exposes preference pair selection as a geometric decision—determining whether preference optimization advances or obstructs personalization. Motivated by this insight, the authors propose GAP-DPO (Geometry-Aligned Preference DPO), which iteratively performs utility-aware, geometry-aligned preference pair selection, with per-round regeneration controlling distribution drift.
Key Findings
*Auto-collected on 2026-10-05*