How Language Models Organize and Structure Moral Knowledge

Forum topic · 小凯 · 2026-08-30

Summary

This paper (arXiv:2608.27402) investigates how large language models organize moral knowledge beyond simple moral-content detection. The authors train six independent linear probes on open-weight language models, one for each Moral Foundations Theory (MFT) category: care/harm, fairness/cheating, liberty/oppression, loyalty/betrayal, authority/subversion, and sanctity/degradation. They then examine how the resulting probe directions relate geometrically in representation space. The key finding is that moral foundation directions neither collapse into a single generic moral detector nor remain fully isolated from one another. Instead, they span a near-maximal number of independent dimensions while sharing a positive common component. This shared component appears to be the signature of moral integration: it is morality-specific relative to matched non-moral concept batteries (mean pairwise cosine similarity of 0.26 versus 0.013). The geometric structure is consistent across architectures and model scales and emerges early in pretraining, reaching its integrated state during the early phases of training. The results suggest LLMs internally represent moral foundations as distinct but related dimensions organized around a common moral signal.

Paper Overview

Research area: NLP
Author: Orion Reblitz-Richardson
Published: 2026-08-27
arXiv: 2608.27402

Abstract

How do large language models (LLMs) organize moral knowledge? Models detect moral content broadly, but detection is a low bar. This work asks whether they go further, distinguishing moral foundations from one another and organizing the relationships between them geometrically.

Method

Six independent linear probes are trained on open-weight language models, one per Moral Foundations Theory (MFT) category:
care/harm
fairness/cheating
liberty/oppression
loyalty/betrayal
authority/subversion
sanctity/degradation
The resulting probe directions are then examined for how they relate to each other in representation space.

Findings

The moral foundation directions neither collapse into a single moral detector nor fully isolate from one another.
Instead, they span a near-maximal number of independent dimensions while sharing a positive common component.
The shared component appears to be the signature of moral integration: it is morality-specific relative to matched non-moral concept batteries (mean pairwise cosine similarity of 0.26 vs 0.013).
This geometric structure is consistent across architectures and model scales.
The integrated state emerges early in pretraining.

Source

arXiv:2608.27402