Moral Entropy: A Bayesian Audit Reveals Systematic Bias in Moral Label Aggregation Rules

> Source: Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment, arXiv:2609.21992 > Author: Maciej Skorski (University of Luxembourg) > Code: https://github.com/maciejskorski/moral-entropy

Imagine a diving competition with three judges. One judge — call him Old Wang — loves giving high scores. Now compare two scoring rules:

• Rule A (any-annotator): an item counts as positive if *any* annotator says so (logical OR).
• Rule B (majority): more than half the annotators must agree.

Rule B sounds more rigorous, but data reveals a counterintuitive result: Rule A amplifies Old Wang's leniency into the whole system, while Rule B filters his inflated scores but also discards the occasionally valid high scores from the other judges. This paper quantifies "Old Wang's bias" within a Bayesian framework and shows that all mainstream aggregation rules for moral labels carry systematic bias — in different directions.

Background: where moral labels come from

Moral Foundations Theory (MFT) divides moral judgment into five dimensions: Care, Fairness, Loyalty, Authority, and Sanctity. A sentence can trigger multiple foundations at once. Corpora are annotated by multiple raters who independently mark which foundations each text triggers, and these annotations are aggregated into a final label.

The problem lies in aggregation. Annotators naturally disagree — sometimes due to genuine moral pluralism (people of different ideologies genuinely see different things), sometimes due to individual bias (some people just over-label Care).

The Moral Entropy framework

Keep the full posterior

Standard pipelines treat disagreement as noise and vote it away. Moral Entropy's core idea: disagreement itself is signal. For each text and foundation, it maintains a full posterior P(label=True | annotations) — e.g., "Care: 0.15, Fairness: 0.78, Loyalty: 0.58" — rather than a hard 0/1 label.

Entropy decomposition: aleatoric vs epistemic

The framework splits the posterior entropy into:

• Aleatoric uncertainty: irreducible disagreement inherent to genuinely ambiguous content.
• Epistemic uncertainty: reducible disagreement from annotator bias, sparse annotation, or noise.

This tells you which disagreements reflect real controversy (to be modeled) and which reflect a broken annotation process (to be fixed).

Per-annotator bias profiles

The framework learns a confusion matrix θ_j for each annotator j on each foundation. When Old Wang labels Care, the framework doesn't take it at face value — it discounts the label by his historical false-positive rate on that foundation.

Auditing existing rules

With the Bayesian posterior as calibrated ground truth, existing rules are audited via cross-entropy, KL divergence, Brier score, and expected calibration error (ECE).

Experimental results

Scale

Three corpora: 106,627 texts, over 250,000 annotations across 15 discourse domains, including the Moral Foundations Twitter Corpus (MFTC) and Moral Foundations Reddit Corpus (MFRC).

any-annotator: inconsistent with the posterior on ~30% of items

The any-annotator rule disagrees with the Bayesian posterior on roughly 30% of items. The direction of disagreement matters:

• At the dataset level, errors are almost all false positives — the rule labels, the posterior says it shouldn't.
• At the foundation level, the picture flips: on MFTC, the false positive rate is 19.9% but the false negative rate is 38.9%.

So the bias direction of any-annotator depends on the granularity of analysis.

majority: misses 63-83% of true positives

Majority and stricter two-vote rules go to the opposite extreme:

• Nearly eliminate false positives (3.5% and 4.2% on MFTC; below 1% on MFRC).
• But miss 63-83% of true positives (mean FNR 63-65% on MFTC, ~83% on MFRC).

Strict rules effectively classify every "contested" item as nonexistent. That isn't rigor — it's throwing out signal along with noise.

Bias direction depends on the rule

One of the paper's most elegant findings: bias is not unidirectional.

• any-annotator biases toward over-labeling (high false positives)
• majority/two-vote biases toward under-labeling (high false negatives)

There is no "better rule" — every rule deviates systematically from truth in a different direction. The Bayesian posterior serves as calibrated ground truth that can audit any rule and quantify how far it is biased, and in which direction.

Soft-label training: 2-3% accuracy gain

Training classifiers on the calibrated soft labels improves accuracy by 2-3% over hard labels. Modest, but "free" — no architecture changes, just better training labels.

Why it matters

1. Voting away disagreement is information destruction. MFT itself predicts that people of different ideologies perceive different moral foundations — voting that disagreement away discards the most valuable signal. 2. A textbook proxy-target trap. "Single labels" are a proxy goal (easy to train on); "preserving disagreement" is the real goal (understanding moral diversity). any-annotator's recall looks great but is full of false positives; majority's precision looks great but misses most true positives. 3. Cardinality-gap in practice. Each annotator's bias is multidimensional — Old Wang may be unreliable on Care but accurate on Fairness. Collapsing annotators to a binary "reliable/unreliable" loses this structure. Keeping the full posterior respects the fact that the true information space is far larger than the label space. 4. Applies to all multi-annotator settings: content moderation, medical imaging diagnosis, sentiment analysis — anywhere multiple raters are collapsed into one label, this framework can audit the aggregation rule's bias.

Limitations

1. The posterior depends on prior assumptions. The paper uses a relatively robust Beta-Binomial conjugate prior, but it is not assumption-free. 2. Confusion matrices need sufficient data per annotator. Sparse annotators yield unstable θ_j estimates; small-sample handling isn't discussed in detail. 3. "Calibrated ground truth" itself needs validation. Independent verification used expert gold labels from eMFD, but expert labels can carry their own bias.

Still, the core contribution stands: aggregation-rule bias can be systematically audited. We previously only knew that "voting loses information"; now we can precisely quantify how much is lost, and in which direction.

Conclusion

The most admirable aspect of this paper is its posture: rather than proposing a better aggregation rule, it proposes a framework for auditing all of them.

It doesn't tell you to use any-annotator or majority. It tells you: whichever rule you use, here is which direction it is biased, and by how much. That is far more mature than "my rule beats yours" — because there are no unbiased rules, only audited biases.

Moral judgment is inherently fuzzy, contested, and person-dependent. Preserving and modeling that fuzziness is more honest — and more useful — than pretending it doesn't exist and voting it away.

Paper: https://arxiv.org/abs/2609.21992 HTML version: https://arxiv.org/html/2609.21992v1 Code: https://github.com/maciejskorski/moral-entropy