Aligning Language Models with General Preferences: Independent Reproduction of Direct Nash Optimization (DNO)

Forum topic · 小凯 · 2026-11-04

Summary

This paper presents an independent reproduction of Direct Nash Optimization (DNO), a method that aligns language models with general preference functions by approximating the Nash equilibrium of a two-player preference game. The authors re-implement the practical variant DNO-Prct on Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, using GPT-4o or same-family open-weight models as annotators. On AlpacaEval 2.0 with GPT-4o as judge, three iterations raise the length-controlled win rate against GPT-4o from 31.7% to 45.3% for Llama and from 6.8% to 18.4% for Mistral, with the tuned 8B model exceeding Llama 3.1 70B Instruct on all reported win rates. Open-weight annotators retain 62-82% of the gains achieved with GPT-4o. Ablations show removing gold responses, data look-back, or teacher-preferred filtering costs 6-10 LC points, while sampling counts and margin sweeps matter less. The paper also discusses pipeline costs and gaps between the practical algorithm and its theoretical foundation.

Aligning Language Models with General Preferences: An Empirical Study

Field: NLP Authors: Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy, Bhuvnesh Kumar, Parivesh Choudhary Published: 2026-11-03 arXiv: 2611.05373

Abstract

Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection, and a DPO-style contrastive update.

The authors re-implement DNO-Prct independently and study it on two open-weight instruct models, Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, with GPT-4o or a larger open-weight model of the same family as the annotator.

Key Findings

Main results: On AlpacaEval 2.0 with GPT-4o as judge, three iterations raise the length-controlled (LC) win rate against GPT-4o from 31.7% to 45.3% for Llama and from 6.8% to 18.4% for Mistral. The tuned 8B model scores above Llama 3.1 70B Instruct on every reported win rate.
Open-weight annotators suffice: With same-family open-weight annotators, runs retain 62% (Llama) and 82% (Mistral) of the LC gain achieved with GPT-4o, showing the rating step does not require a proprietary model.
Ablations: Variants removing the gold responses, the data look-back, or accepting only teacher-preferred pairs score 6 to 10 LC points below the full procedure. Varying on-policy samples per prompt (1 to 7) moves LC win rate by at most 2.1 points; a margin sweep (1 to 5) by at most 2.4 points.
Caveats: All results are single runs, and GPT-4o serves as both teacher and evaluation judge; the paper discusses implications for the reported gains. The authors also report pipeline costs and where the practical algorithm departs from the theory it approximates.