Moral Maps: How LLMs Organize Moral Knowledge into Structured Geometry

Paper: How Language Models Organize and Structure Moral Knowledge Author: Orion Reblitz-Richardson arXiv: 2608.27402 Published: 2026-08-27

The Judge's Dilemma

The post opens with a courtroom analogy: a judge weighing a poor man who stole medicine to save his child against an aristocrat who tortured animals for fun. Human moral hesitation arises because morality is not black-and-white—it is a complex, multidimensional, sometimes self-contradictory structure. The paper asks: do large language models (LLMs) merely detect moral-sounding words, or do they internally build a genuine "map" of good and evil?

Moral Foundations Theory (MFT)

The analysis is grounded in Jonathan Haidt's Moral Foundations Theory, which posits six psychological pillars of human moral judgment:

Individualizing foundations (protecting individuals, fairness): 1. Care/Harm 2. Fairness/Cheating 3. Liberty/Oppression

Binding foundations (group cohesion, order): 4. Loyalty/Betrayal 5. Authority/Subversion 6. Sanctity/Degradation

MFT predicts these six foundations form two structurally distinct clusters—a prediction validated in cross-cultural human surveys.

Method: Linear Probing

The researchers trained linear probes at every layer of three open models (OLMo-2 1B, OLMoE-1B-7B mixture-of-experts, OLMo-2 7B), one probe per moral foundation, distinguishing foundation-related sentences from neutral ones. Each probe's weight vector defines a "moral direction" in the model's representation space.

Key Findings

Integration geometry, not collapse or isolation

All six foundations are decodable at near-100% accuracy—but the paper's core contribution is the geometric analysis. Three possible states were defined:

| State | Pairwise cosine similarity | Meaning | |---|---|---| | Collapse | ~1 | All foundations merge into one "moral salience" axis | | Isolation | ~0 | Foundations are fully independent slots | | Integration | Structured, between 0 and 1 | Distinct but related dimensions sharing a common component |

LLMs exhibit Integration:

• Average pairwise cosine similarity of 0.22–0.27 (e.g., 0.262 at layer 7 of OLMo-2 1B)
• Effective dimensionality of 5—the maximum possible for six mean-centered directions—ruling out collapse into a lower-dimensional subspace

A morality-specific shared component

Moral directions share a positive common component. A matched control using non-moral concepts (tools, emotions, spatial relations) showed:

• Moral pairs: average cosine 0.26
• Non-moral control pairs: 0.013

A nearly 20x gap, showing LLMs build a dedicated moral representation space where foundations are both independent and interconnected.

Very early emergence

The moral geometry approaches its final integrated state extremely early in pretraining—before probe accuracy even saturates. Moral structure appears to be an early emergent property of language learning: models acquire value orientation alongside word meaning. The structure is robust across scale (1B–7B), architecture (dense vs. MoE), and layers.

MFT's prediction vs. corpus statistics

Existing evidence does not support MFT's individualizing/binding split in LLM geometry. Instead, the structure appears to reflect statistical co-occurrence patterns in the training corpus. This raises a philosophical question: are LLMs discovering deep moral structure, or merely reproducing textual correlations? The evidence favors the latter.

Moral dilemmas: compositionality plus conflict

On moral dilemmas (e.g., trolley problems), the study finds:

• Dilemma directions are partly composed of their constituent foundation directions, with composition strength 2.7x mismatch baselines—models genuinely co-activate, say, "care" and "fairness" when facing a dilemma
• Most of the dilemma direction's variance encodes conflict-specific structure: the model represents the *tension itself*, not just a superposition of foundations

Implications

1. Terrain, not rules: LLM moral knowledge is a structured, navigable high-dimensional space rather than a set of hardcoded rules or a simple good/bad classifier. 2. Detection is easy; understanding is hard: near-perfect detection is a low bar; the observed structure may stem from corpus statistics rather than deep psychological understanding. 3. Early emergence as a warning: moral bias may solidify before models speak fluently—value interventions may need to happen very early in training.

Future Directions

Reading an LLM's moral geometry could enable bias diagnosis, value steering by adjusting geometric relations, prediction of unstable judgments in dilemmas, and cross-cultural comparison of models trained on different languages.

References

• Reblitz-Richardson, O. (2026). How Language Models Organize and Structure Moral Knowledge. *arXiv preprint arXiv:2608.27402*.
• Haidt, J., & Graham, J. (2007). When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize. *Social Justice Research*, 20(1), 98-116.