[论文] Aligning Language Models with General Preferences: An Empirical Study ...

论文概要

研究领域: NLP
作者: Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy, Bhuvnesh Kumar, Parivesh Choudhary
发布时间: 2026-11-03
arXiv: 2611.05373

中文摘要

Direct Nash Optimization (DNO) 通过对二人偏好博弈的纳什均衡进行逼近来对齐语言模型与一般偏好函数。其实用变体 DNO-Prct 迭代执行：策略上采样、LLM 对样本与教师金标回答打分、大间隔偏好对筛选，以及 DPO 式对比更新。我们在两个开源指令模型 Llama 3.1 8B Instruct 和 Mistral 7B Instruct v0.1 上独立复现了 DNO-Prct，以 GPT-4o 或同家族更大的开源模型作为标注器。在 AlpacaEval 2.0 上（GPT-4o 当评判），三轮迭代把对 GPT-4o 的长度受控（LC）胜率从 31.7% 提升到 45.3%（Llama）、从 6.8% 提升到 18.4%（Mistral），且微调后的 8B 模型在所有已报告胜率上超过 Llama 3.1 70B Instruct。使用同家族开源标注器时，仍保留了 GPT-4o 标注带来 LC 增益的 62%（Llama）和 82%（Mistral），说明打分步骤并不依赖闭源模型。消融显示：去掉金标、去掉数据回溯、或只接受教师偏好对，都会比完整流程低 610 个 LC 点；而每提示采样数（17）与间隔扫描（1~5）的影响分别不超过 2.1 和 2.4 个点。我们也报告了流水线的成本，以及实用算法在何处偏离了它逼近的理论。

原文摘要

Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update. We re-implement DNO-Prct independently and study it on two open-weight instruct models, Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, with GPT-4o or a larger open-weight model of the same family as the annotator. On AlpacaEval 2.0 with GPT-4o as judge, three iterations raise the length-controlled (LC) win rate against GPT-4o from 31.7% to 45.3% for Llama and from 6.8% to 18.4% for Mistral, and the tuned 8B model scores above Llama 3.1 70B Instruct on every reported win rate. With same-family open-weight annotators, the runs retain 62% (Llama) and 82% (Mistral) of the LC gain achieved with GPT-4o as annotator, so the rating step does not require a proprietary model. In a separate ablation configuration, variants that remove the gold responses, remove the data look-back, or accept only teacher-preferred pairs score 6 to 10 LC points below the full procedure, whereas varying the number of on-policy samples per prompt (1 to 7) moves the LC win rate by at most 2.1 points, and a margin sweep (1 to 5) run in its own pipeline by at most 2.4 points. We also report the cost of the pipeline and where the practical algorithm departs from the theory it approximates. All results are single runs, and GPT-4o is both the teacher and the evaluation judge; we discuss what this implies for the reported gains.

自动采集于 2026-11-05

#论文 #arXiv #AI #小凯