Participatory Moral AI Is Not Neutral: How Developer Choices Shape Preference Elicitation

Forum topic · 小凯 · 2026-08-18

Summary

A study by Taenyun Kim, Edyta Bogucka, and Daniele Quercia (arXiv:2508.08540) examines how developer decisions in moral AI preference elicitation silently shape aggregated ethical preferences. In a two-stage study (N=809) across three deployment scenarios—AI kidney allocation, AI agents simulating absent workers, and generative AI depictions of deceased people—the authors analyze three pipeline stages: feature scoping, voter sampling, and question framing. They find that morally relevant features shift across scenarios, so feature patterns should not be assumed transferable across domains; roughly one-third of feature preferences vary by political ideology, with some differences even reversing direction, meaning the ideological composition of the voter pool affects aggregated preferences; and question wording can shrink or widen ideological gaps by up to a full scale point, while framing changes how moral foundations relate to judgments. The findings argue that vote-based alignment cannot deliver fair or transparent AI through aggregation alone; every stage of the moral AI elicitation pipeline should be audited and disclosed.

Participatory Moral AI Is Not Neutral: The Invisible Hand of Developer Choices

Research area: ML Authors: Taenyun Kim, Edyta Bogucka, Daniele Quercia arXiv: 2508.08540

Overview

As AI systems make more morally loaded decisions across society, one response has been moral preference elicitation: researchers poll participants on hypothetical dilemmas and use the aggregated votes to train a policy that an AI model then applies at scale.

However, before any vote is cast, developers make three key choices in the moral AI elicitation pipeline:

Feature scoping — which features go to a vote
Voter sampling — which voters to include
Question framing — how the question is presented

These choices are often opaque, undocumented, and treated as technical details rather than normative ones.

Findings

The authors examined each of these choices in a two-stage study (N=809) covering three deployment scenarios: AI kidney allocation, AI agents simulating absent workers, and generative AI depictions of deceased people.

1. Feature scoping: Morally relevant features shift across scenarios, suggesting that feature patterns should not be assumed to transfer across deployment domains. 2. Voter sampling: Roughly one-third of feature preferences vary with political ideology, and some differences even reverse direction. The ideological composition of the voter pool therefore shapes the final aggregated preference profile. 3. Question framing: The wording of elicitation questions can shrink or widen ideological gaps by up to a full scale point, and framing conditions also change how moral foundations relate to participants' judgments.

Conclusion

Taken together, these findings show that vote-based alignment cannot achieve fair or transparent AI through aggregation alone. At minimum, every stage of the moral AI elicitation pipeline should be audited and disclosed.

Source: arXiv:2508.08540