ComPO: A Zeroth-Order Paradigm for LLM Preference Alignment

Forum topic · 小凯 · 2026-09-18

Summary

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences due to their computational and memory efficiency, but the likelihood displacement phenomenon motivates new ways to extract information from preference pairs with small likelihood margins. This paper introduces Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method built on comparison oracles that extracts directional information from such preference pairs without directly optimizing a differentiable preference loss on them. The authors establish convergence guarantees for the basic offline scheme under smoothness, gradient sparsity, and oracle-objective compatibility conditions. They further propose online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations to achieve reverse KL control relative to a reference policy. Performance guarantees are derived for the constrained scheme under local coverage and in-distribution pairwise reward accuracy conditions. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 show the method outperforms existing direct alignment methods, including in length-controlled win rate, with pairwise-level diagnostic evidence consistent with mitigated likelihood displacement. Paper: arXiv 2609.19144.

Paper Overview

Research area: NLP Authors: Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin Published: 2026-09-16 arXiv: 2609.19144

Abstract (translated)

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. This paper proposes and analyzes Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them.

The authors establish a convergence guarantee for the basic offline scheme under smoothness, gradient sparsity, and compatibility conditions between the oracle and a latent objective. They further introduce online ComPO, which retains the offline comparison mechanism and leverages unlabeled policy generations to achieve reverse KL control relative to a reference policy. Along the theoretical thread of preference fine-tuning, performance guarantees for the basic constrained scheme are established under local coverage and in-distribution pairwise reward accuracy conditions.

Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 show that the method outperforms existing direct alignment methods (including length-controlled win rate), and pairwise-level diagnostic evidence is consistent with the inference that likelihood displacement is mitigated.

Key Contributions

Zeroth-order alignment: uses comparison oracles instead of optimizing a differentiable preference loss directly on preference pairs with small likelihood margins.
Offline scheme convergence guarantee: proven under smoothness, gradient sparsity, and oracle-objective compatibility conditions.
Online ComPO: keeps the offline comparison mechanism and uses unlabeled policy generations for reverse KL control against a reference policy, with performance guarantees under local coverage and in-distribution pairwise reward accuracy assumptions.
Empirical validation: outperforms existing direct alignment methods on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3, including length-controlled win rate; diagnostics suggest reduced likelihood displacement.

Link

arXiv: 2609.19144

--- *Auto-collected on 2026-09-18*