Which Objectives Need a Dial? Predicting Objective Conflict and Coverage in MODPO

Forum topic · 小凯 · 2026-09-25

Summary

This paper by David Tsoi and Esra Dönmez (arXiv:2609.26929) examines pluralistic alignment through Multi-Objective Direct Preference Optimization (MODPO), which uses objective weights to span a continuum of trade-offs between competing goals. The authors address two questions: when can a single model improve two objectives simultaneously, and how can many trade-offs be covered without training a separate model for each? Across seven objective pairs from HelpSteer and UltraFeedback, two pre-training measurements predict whether objectives align or conflict on human-annotated data, but fail on AI-annotated data, where response length and repetition confound reward-model scores. For broader trade-off coverage, selecting the nearest already-trained model and merging model parameters both help, but neither consistently matches direct training. The findings offer practical guidance for building steerable models serving diverse human preferences.

Paper Overview

Field: NLP
Authors: David Tsoi, Esra Dönmez
Published: 2026-09-25
arXiv: 2609.26929

Abstract (Original)

People hold diverse, sometimes conflicting values, so no single aligned model can satisfy everyone. Pluralistic alignment therefore calls for steerable models that can balance competing objectives differently. Multi-Objective Direct Preference Optimization (MODPO) does this by using an objective weight to span a continuum of trade-offs. The paper studies two questions: when can one model improve two objectives simultaneously, and how can many trade-offs be covered without training a separate model for each?

Key Points

Setup: Seven objective pairs drawn from HelpSteer and UltraFeedback are evaluated under MODPO, which spans a continuum of trade-offs via an objective weight.
Predicting conflict: Two pre-training measurements can predict whether a pair of objectives will align or conflict for human-annotated data.
AI-annotated data caveat: The same predictors fail for AI-annotated data, where response length and repetition confound reward-model scores.
Trade-off coverage: Selecting the nearest already-trained model and merging model parameters both help cover a broader range of trade-offs, but neither approach consistently matches directly training for each trade-off.
Takeaway: Practical guidance for building steerable models that serve pluralistic preferences, including when multi-objective improvement is feasible and when per-objective training remains necessary.

---

*Auto-collected on 2026-09-25.*