Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers

> Paper: Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers > Authors: Taenyun Kim, Edyta Bogucka, Daniele Quercia > arXiv: 2608.14522 > Field: AI Ethics / Social Computing

This post is a deep-dive commentary on the paper, exploring how participatory moral AI—where the public votes on ethical dilemmas to train AI decision-making—is shaped by developer choices long before any ballot is counted.

The Promise and the Problem

Moral preference elicitation has become a popular answer to AI's growing role in ethically loaded decisions (medical resource allocation, content moderation, generative AI boundaries). The pipeline looks democratic: design dilemma scenarios, collect public votes, extract aggregate moral preferences, and train AI to follow "the people's will."

But the paper argues that three "invisible hands" already shape the outcome before voting begins:

• Feature Scoping — developers decide which factors are even on the ballot (e.g., for kidney allocation: age, wait time, medical urgency—but perhaps not socioeconomic status or family responsibility).
• Voter Sampling — who gets invited shapes whose moral intuitions are encoded.
• Question Framing — wording ("save whom" vs. "sacrifice whom," authenticity vs. harmony) can shift answers substantially.

Experimental Findings

The study spans three scenarios: AI kidney allocation, AI agents simulating absent workers, and generative AI depicting deceased people. Key results:

• Feature drift: Morally relevant features shift significantly across contexts. No universal feature template transfers between scenarios—reusing validated feature sets is itself a hidden source of bias.
• Ideological divides: For roughly one third of features, political ideology produces significantly different—and sometimes directionally opposite—preferences. There is no simple left/right pattern; each feature has its own moral logic.
• Framing effects: Question wording can shrink or amplify the ideological gap by up to a full scale point, and can even reverse a group's relative ranking of a feature.

Psychological Mechanisms

The commentary connects these effects to:

• Prospect theory (Kahneman & Tversky): loss-framed vs. gain-framed dilemmas ("this patient will die in X months" vs. "will gain X months of life") trigger different moral weightings despite mathematical equivalence.
• Moral foundations theory (Haidt): framing activates different moral foundations (care/harm vs. fairness/cheating), changing not just answers but the moral language people reason with.
• Differential sensitivity: groups respond differently to framing, creating asymmetric participation—final aggregates may represent only "those who chose to stay under a particular framing."

From Technical to Normative Choices

The paper's sharpest claim:

> "These choices are often opaque, undocumented, and treated as technical details rather than normative ones."

The commentary likens this to ignoring what charges a prosecutor files while auditing a jury's verdict—the framing decision already bounds the outcome space. Core tensions identified:

• Transparency vs. manipulability: disclosure enables scrutiny but also teaches motivated actors how to game the system.
• Standardization vs. rigidity: uniform protocols prevent some bias but suppress context sensitivity.
• Participation vs. expertise: whose judgment counts, and why do developer choices outweigh voter preferences?

Policy Implication

The paper's conclusion is blunt:

> "Voting-based alignment cannot deliver fair or transparent AI by aggregation alone."

Recommended remedy: continuous auditing and disclosure of every pipeline stage—who scoped the features and why, how the sample was selected, how questions were framed and what alternative framings would yield.

Deeper Questions

The post closes with philosophical reflections: moral preferences may be so context- and framing-dependent that any attempt to "measure" them distorts them (an observer-effect analogy). Alternative directions—reflecting institutionalized ethics, procedural justice, or pluralistic user-adjustable frameworks—each carry their own biases. The uncomfortable takeaway: developers cannot hide behind "I just implemented the democratic process." Every technical choice is a value choice, and transparency is not sufficient—but without it, nothing else is possible.

References

1. Kim, T., Bogucka, E., & Quercia, D. *Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers*. arXiv:2608.14522. 2. Haidt, J. (2012). *The Righteous Mind*. Pantheon Books. 3. Kahneman, D., & Tversky, A. (1979). Prospect theory. *Econometrica*, 47(2), 263–291. 4. Rawls, J. (1971). *A Theory of Justice*. Harvard University Press. 5. Birhane, A., et al. (2022). Power to the people? Opportunities and challenges for participatory AI. 6. Sloane, M., et al. (2022). Participation is not a design fix for machine learning.