Perplexity, Semantic Entropy, and How Minds Learn at the Edge of Chaos

Forum topic · 小凯 · 2026-02-02

Summary

This speculative essay explores perplexity and semantic entropy as unifying measures of uncertainty across brains, AI models, and civilizations. Perplexity is defined via the geometric-mean inverse prediction probability of a sequence, linking information theory to neural prediction-error signals in the anterior cingulate cortex, and to LLM evaluation where a 'perplexity paradox' can emerge during reinforcement-learning fine-tuning. Semantic entropy is analyzed at token, planning-token, and conceptual-topology levels, showing a two-phase learning pattern: early entropy collapse and skill consolidation, followed by rising semantic diversity tied to reasoning gains. The author proposes a stochastic differential equation framing cognition as motion in a potential field, and argues that religious systems function as cognitive control mechanisms—absorbing doctrinal anomalies, encoding humility and obedience, and institutionally suppressing rebellious transitions—locking societies into low-entropy stable states. Civilizational learning, the essay suggests, depends on the statistical distribution of intolerance of uncertainty, with innovation maximal at moderate tolerance and near the 'edge of chaos' between order and disorder. References cite arXiv preprints and behavioral/neuroscience literature; the piece is theoretical synthesis rather than empirical result.

Perplexity Wells: How Minds Learn to Fly in the Abyss of Uncertainty

*An English translation of a speculative essay from zhichai.net connecting information theory, cognitive neuroscience, LLM training dynamics, and the sociology of religion.*

Perplexity: Both Poison and Remedy

Perplexity (PPL) quantifies a model's — or a mind's — uncertainty about what comes next. It is defined as the geometric-mean inverse of sequence prediction probability:

PPL = \exp\left(-\frac{1}{N}\sum_{i=1}^{N}\log P(w_i|w_{<i})\right)

Intuitively: if PPL = 100, the system is effectively guessing each next word from 100 equally plausible candidates. Here $$N$$ is sequence length and $P(w_i|w_{<i})$ is the conditional probability of the current word given its context.

In cognitive neuroscience, perplexity maps onto the brain's prediction-error signal: a truly surprising image strongly activates the anterior cingulate cortex, with theta-band power spiking. Neuroaesthetics experiments suggest moderate perplexity yields the highest aesthetic pleasure — too familiar feels bland, too alien feels incomprehensible.

In AI, perplexity is the standard thermometer for language models (lower is better), yet a "perplexity paradox" can appear during RL fine-tuning: perplexity on answers keeps falling (memorization strengthens) while perplexity on questions rises (understanding degrades) — like a student who memorizes exam points without grasping why.

Semantic Entropy: The Breathing Rhythm of Thought

Where perplexity measures surprise at the next moment, semantic entropy measures the depth of a system's exploration of meaning space, across three layers:

Micro: token-level Shannon entropy $H(X) = -\sum p(x_i)\log p(x_i)$ , which "collapses" late in training as basic skills (arithmetic, formatting) become machine-like and deterministic.
Meso: diversity of planning-type tokens (e.g., "let's try another approach"), which keeps rising through a second phase in Qwen and Llama models, tracking reasoning accuracy.
Macro: topological entropy of concept space — whether a mind is a closed castle or an open sky.

Phase 1 resembles a craftsman mastering tools (token entropy plummets); phase 2 resembles a philosopher inventing new strategies (semantic entropy soars). Human learning follows the same arc: first walking (low-entropy automation), then dancing (high-entropy creation).

Tolerance for Perplexity

Psychology measures the inverse trait with the Intolerance of Uncertainty Scale (IUS-12): higher scores predict anxiety and poorer cognitive function in breast cancer patients, and academic stress via rumination in high schoolers. Religious systems, the author argues, systematically raise domain-specific tolerance thresholds, converting extreme perplexity about death, suffering, and injustice into the low-entropy comfort of a "divine order."

A Unified Dynamical Field

All cognitive systems are modeled by one stochastic differential equation:

\frac{dx}{dt} = -\Gamma \frac{\delta \Phi}{\delta x} + \sqrt{2\Gamma T}\, \xi(t) + R(x, t)

Term 1 is gravity, pulling thought toward familiar valleys of the potential field $\Phi(x)$ .
Term 2 is noise, waves delivering random perturbations.
Term 3, $$R(x,t)$$ , is a non-conservative re-injection flow — analogous to self-attention — recycling information across layers for iterative reasoning.

Learning sculpts $\Phi(x)$ so that correct reasoning trajectories rest in wide, flat basins. Religion additionally builds deep "gravitational wells" and tall "cognitive walls" that keep thought near doctrine.

Two-Phase Learning and Religious Suppression

Both humans and large models show: Phase 1 — perplexity plunges, token entropy collapses, skills consolidate; Phase 2 — semantic entropy rises, planning-token diversity surges, innovation peaks at the edge of chaos. The phase transition is preceded by a marked surprise spike, like a pressure drop before a storm.

The essay identifies three mechanisms by which religion dampens rebellious cognitive transitions:

1. Doctrinal absorption: re-encoding high-perplexity events into sacred narrative, compressing semantic entropy. 2. Moral encoding: humility, obedience, endurance raise tolerance thresholds. 3. Institutional suppression: ritual, taboo, authority structures, heresy trials — the cognitive wall.

Medieval Europe's Inquisition and imperial China's examination–Confucian system are presented as dynamically isomorphic: both lock society into low-semantic-entropy stable states away from the edge of chaos.

Civilizational Learning

Civilizational learning capacity is framed as an emergent property of the statistical distribution of intolerance of uncertainty: thick-tailed distributions yield innovation bursts; thin tails yield stability without progress. Innovation rate follows an inverted-U against religious tolerance — moderate (controlled) perplexity is optimal. Early modern Europe's cascading phase transitions — the Reformation lowering tolerance thresholds, the Scientific Revolution shifting perplexity domains, the Enlightenment rebuilding the cognitive landscape — illustrate the curve.

The P–S Phase Space

In perplexity–semantic-entropy (P–S) space:

Religious systems: low P, low S (dogmatic steady state)
Scientific exploration: high P, high S (innovation zone)
Optimal learning: moderately high, at the edge of chaos

| Trajectory | Perplexity | Semantic entropy | Cognitive analogue | |---|---|---|---| | Convergent | monotone decreasing | monotone decreasing | rote memorization | | Oscillatory | periodic fluctuation | periodic fluctuation | theological debate | | Chaotic | irregular fluctuation | high and volatile | creative exploration | | Transitional | abrupt drop | rise then fall | insight, paradigm shift |

Closing

The essay acknowledges limits — quantum cognition effects, unpredictability of superhuman intelligence, group heterogeneity — but suggests that through computational-theology experiments, agent-based modeling, and historical backtesting, we might learn to dance gracefully in perplexity rather than flee it. When humans, machines, and civilizations collectively learn to dance at the edge of chaos, that may be the prelude to the next great leap.

References

1. Unified Dynamical Field Theory of Cognition (2024). arXiv:2601.10221 2. Hierarchical Reasoner: Direct Learning of Planning from Entropy (2024). Tiger-AI-Lab 3. Semantic Entropy and Phase Transitions in Large Language Models (2024). arXiv:2509.03646 4. Religious Systems as Cognitive Control Mechanisms (2023). Humanities and Social Sciences Communications 5. The Edge of Chaos in Cognitive Dynamics (2023). Nature Human Behaviour

*Note: this is a theoretical/speculative synthesis; cited preprints and claims have not been independently verified by the editor.*