English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

A Zeroth-Order Paradigm for LLM Preference Alignment: Comparison-based Preference Optimization (ComPO)

Forum topic · 小凯 · 2026-09-18

Summary

This post introduces ComPO (Comparison-based Preference Optimization), a zeroth-order alignment method for large language models based on comparison oracles, presented in arXiv paper 2609.19144. Motivated by the likelihood displacement problem in direct preference alignment, ComPO extracts directional information from preference pairs with small likelihood margins without optimizing a differentiable preference loss on them. The authors establish convergence guarantees for the offline scheme under smoothness, gradient sparsity, and oracle-objective compatibility conditions. They also propose online ComPO, which uses unlabeled policy generations to control reverse KL divergence against a reference policy, with performance guarantees under local coverage and in-distribution pairwise reward accuracy assumptions. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 show improvements over existing direct alignment methods, including length-controlled win rates, with pairwise-level diagnostics consistent with mitigated likelihood displacement.

Paper Overview

Field: NLP Authors: Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin Published: 2026-09-16 arXiv: 2609.19144

Summary

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins.

This paper proposes and analyzes Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these preference pairs without directly optimizing a differentiable preference loss on them.

Key Contributions

  • Convergence guarantees for the basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective.
  • Online ComPO: retains the offline comparison mechanism while leveraging unlabeled policy generations to achieve reverse KL control relative to a reference policy. Performance guarantees are established under local coverage and in-distribution pairwise reward accuracy conditions.
  • Empirical results: experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 show the method outperforms existing direct alignment methods (including length-controlled win rates), with pairwise-level diagnostic evidence consistent with mitigating likelihood displacement.

Original Abstract (excerpt)

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offli...

--- *Auto-collected on 2026-09-18*

Tags

#llm#preference-alignment#zeroth-order-optimization#compo#reinforcement-learning-from-human-feedback#arxiv#nlp

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178634940