Paper Overview
Field: ML Authors: Taenyun Kim, Edyta Bogucka, Daniele Quercia Published: 2026-08-17 arXiv: 2508.08540
Summary
As AI systems make more morally loaded decisions across society, one response has been moral preference elicitation: researchers poll participants on hypothetical dilemmas and use the aggregated votes to train a policy that an AI model then applies at scale. However, before any vote is cast, developers make three key choices in the moral AI elicitation pipeline: feature scoping, voter sampling, and question framing. In other words, they decide which features go to a vote, which voters to include, and how to present the question. These choices are often opaque, undocumented, and treated as technical details rather than normative decisions.
The authors examine each of these choices within a common empirical study and show that each can shape the preferences produced by moral AI elicitation. In a two-stage study (N=809) and three deployment scenarios (AI kidney allocation, simulated absentee-worker AI agents, and generative AI depictions of deceased people), they examine three main stages of the pipeline:
1. Feature scoping: Morally relevant features change across scenarios, suggesting feature patterns should not be assumed to transfer across deployment domains. 2. Voter sampling: Roughly one-third of feature preferences vary by political ideology, and some differences run in opposite directions—so the ideological composition of the voter pool influences the final aggregated preference profile. 3. Question framing: The wording of elicitation questions can shrink or widen ideological gaps by up to a full scale point, and framing conditions change how moral foundations relate to participants' judgments.
Taken together, these findings show that vote-based alignment cannot achieve fair or transparent AI through aggregation alone. At minimum, every stage of the moral AI elicitation pipeline should be audited and disclosed.
Original Abstract
As AI systems make more morally loaded decisions across society, one response has been moral preference elicitation. In this approach, researchers poll participants on hypothetical dilemmas and use the aggregated votes to train a policy that an AI model then applies at scale. Before any vote is cast, developers make three key choices in the moral AI elicitation pipeline: feature scoping, voter sampling, and question framing. In other words, they decide which features go to a vote, which voters to include, and how to present the question. These choices are often opaque, undocumented, and treated as technical details rather than normative ones. We examine each of these choices within a common empirical study and show that each can shape the preferences produced by moral AI elicitation...
--- *Auto-collected on 2026-08-18*