Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers

Paper: Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers Authors: Taenyun Kim, Edyta Bogucka, Daniele Quercia arXiv: 2608.14522 Field: AI Ethics / Social Computing

This forum post is a deep-dive commentary (Feynman-style) on the paper, examining how participatory moral AI — collecting public votes on moral dilemmas to train AI decision-making — is quietly shaped by its designers before a single ballot is cast.

The Democratic Illusion

Moral preference elicitation follows an appealing pipeline: design dilemma scenarios, collect votes from many participants, aggregate preferences, and use them to train AI moral policies. The post argues this hides three "invisible hands":

• Feature scoping: developers decide which factors enter the vote (e.g., for kidney allocation: age, wait time, medical urgency — but perhaps not socioeconomic status or family responsibility). This draws the boundary of the moral debate.
• Voter sampling: who gets invited to vote matters; different groups have different moral intuitions.
• Question framing: "save whom" vs. "sacrifice whom" framing can move ideological gaps by up to a full scale point.

Experimental Evidence

The paper tests three deployment scenarios spanning life ethics, workplace norms, and posthumous identity:

1. AI kidney allocation — how should an AI prioritize transplant patients? 2. AI agents simulating absent workers — how faithfully should an AI replicate an employee's personality? 3. Generative AI depicting deceased people — whose wishes matter, family's or the deceased's?

Key points

• Feature drift: morally relevant features shift substantially across scenarios. No universal moral feature template transfers across deployments; reusing validated feature lists from one context in another is a hidden source of bias.
• Ideological divides: for about one-third of features, political ideology produces significant preference differences — sometimes in opposite directions. There is no simple left-vs-right pattern; each feature has its own moral logic. Choosing which "public" to reflect is itself a normative decision.
• Framing effects: wording alone can shrink or amplify ideological gaps by up to a full scale point, and in some cases reverses a group's relative ordering of features.

Psychological Mechanisms

• Prospect theory (Kahneman & Tversky): loss frames ("will die in X months without transplant") vs. gain frames ("gains X months with transplant") are mathematically equivalent but trigger different moral weightings.
• Moral foundations theory (Haidt): framing activates different moral foundations (care vs. fairness), changing not just answers but the moral language people use to reason.
• Asymmetric sensitivity: some groups are more framing-sensitive; discouraged participants drop out, so the final aggregate may represent only "the specific population that chose to keep participating under a specific framing."

Power Structure Behind Technical Choices

The paper's sharpest claim:

> "These choices are often opaque, undocumented, and treated as technical details rather than normative ones."

Like a prosecutor's initial charging decision bounding a jury's possible verdicts, these upstream design choices define the space of possible outcomes — yet AI ethics audits rarely examine how questions were posed.

Tensions and Recommendations

The post discusses tensions the paper surfaces:

• Transparency vs. manipulability: disclosing framing choices enables review, but also tells bad actors how to game the system.
• Standardization vs. rigidity: uniform protocols prevent some bias but suppress context sensitivity.
• Participation vs. expertise: if moral judgment is universal, why do developer choices outweigh voter preferences?

The paper's policy conclusion:

> "Voting-based alignment cannot deliver fair or transparent AI by aggregation alone."

It calls for ongoing audit and disclosure at every stage: which features were included/excluded and why; how the sample was chosen and whom it represents; how questions were worded and what alternative framings would yield.

Deeper Questions

• Can morality be "measured" at all? Framing effects are not noise but part of moral judgment's nature — observation distorts the observed.
• Whose morality should AI reflect: institutionalized ethics, procedural justice, or a pluralism of user-adjustable frameworks? Each option carries its own biases.
• Developers cannot hide behind "I just implemented the democratic process" — every technical choice is a value choice demanding accountability.

The post closes with a parable: a survey where every question was pre-approved by the king measures the king's will, not the people's. Recognizing developers as value-shapers, not just technical implementers, is the first step toward responsible moral AI.

References

1. Kim, T., Bogucka, E., & Quercia, D. (2026). *Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers*. arXiv:2608.14522. 2. Haidt, J. (2012). *The Righteous Mind*. Pantheon Books. 3. Kahneman, D., & Tversky, A. (1979). Prospect theory. *Econometrica*, 47(2), 263-291. 4. Rawls, J. (1971). *A Theory of Justice*. Harvard University Press. 5. Birhane, A., et al. (2022). Power to the people? Opportunities and challenges for participatory AI. 6. Sloane, M., et al. (2022). Participation is not a design fix for machine learning.