Information Is Not a Physical Quantity: What Epiplexity Reveals About What AI Extracts from Data

> Paper analyzed: arXiv 2601.03220 — *From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence* > Authors: Marc Finzi*, Shikai Qiu*, Yiding Jiang*, Pavel Izmailov, J. Zico Kolter, Andrew Gordon Wilson > Affiliations: Carnegie Mellon University, New York University > Published: 2026-01-06 (v2: 2026-03-16)

An Uncomfortable Question

How much information did AlphaZero learn from data? By Shannon information theory and Kolmogorov complexity, the answer is zero: its only inputs are the rules of chess and a self-play algorithm—both minimal programs. Deterministic processes cannot create information; that's the data processing inequality (DPI). Yet AlphaZero's weights clearly encode opening strategy, middlegame tactics, and endgame patterns that don't exist in its raw inputs.

The paper's starting point: classical information theory assumes observers with unlimited computation, but real AI systems (and humans) are computationally bounded. It proposes epiplexity (from epistemic complexity), measuring the structured information a computationally bounded observer can extract from data.

Three Paradoxes of Classical Information Theory

1. Deterministic transformations cannot add information. Yet AlphaZero learns superhuman policy from self-play; CSPRNGs stretch a short seed into 'random-looking' output; simple cellular automata produce complex patterns. Under computational bounds, deterministic transformations *can* increase time-bounded information. CSPRNG output has near-maximal entropy for polynomial-time observers but near-zero epiplexity (no learnable structure), whereas ECA Rule 54 (Class IV) produces both randomness and structure.

2. Information is order-invariant. Yet language models model text far better left-to-right, and one-way functions are easy forward, hard backward. Experiments on chess show that training on reversed order (board → move sequence) yields higher epiplexity and better downstream OOD performance (centipawn evaluation). Theorem 13: assuming one-way permutations exist, forward vs. reversed time-bounded entropy can differ by ω(log n).

3. Likelihood modeling is just distribution matching. But to predict masked ECA inputs, models must learn to enumerate possible inputs and verify them through the generative function—a far more complex program than the generator itself. Computationally bounded predictors must learn programs *more complex* than the generating process.

Formal Definition

Given X on {0,1}^n with compute limit T(n), the optimal program minimizes time-bounded MDL:

P* = argmin_{P ∈ P_T} { |P| + E[log 1/P(X)] }

• **Epiplexity S_T(X) := |P*| — program length, i.e., structural information
• Time-bounded entropy H_T(X) := E[log 1/P*(X)] — residual unpredictability

The structure/randomness boundary depends on the observer's compute budget: CSPRNG output is 'simple' for unbounded observers but 'random' for polynomial-time ones.

Measuring Epiplexity

• Prequential coding: cumulative prediction loss during training; epiplexity ≈ area under the loss curve above final loss. Cheap, slightly less rigorous.
• Requential coding: teacher–student framework; encoding cost per step ≈ KL(P^t_i || P^s_i). Rigorous but 2–10× more expensive.

The two rankings agree closely, so prequential coding suffices in practice.

Empirical Findings

• ECA: Rule 15 (periodic) — low entropy, low epiplexity. Rule 30 (chaotic, believed one-way) — maximal entropy, near-zero epiplexity. Rule 54 (Class IV) — high epiplexity with moderate entropy. Similar programs yield vastly different information profiles.
• Chess ordering: reversed order forces richer board-state representations, giving higher epiplexity and significantly better OOD centipawn performance.
• Natural data (5B tokens, 6×10^18 FLOPs): OpenWebText has highest epiplexity, then chess; CIFAR-5M images have the most total information but lowest epiplexity (99%+ is random pixel noise). Extrapolating scaling laws to 1T tokens / 10^25 FLOPs, language retains the highest epiplexity; VQ tokenization greatly improves image epiplexity; video has high temporal redundancy and even lower epiplexity than images. Text pretraining transfers best not because it carries the most bits, but because its structured-information density is highest.
• Data selection**: Adaptive Data Optimization (ADO, Jiang et al. 2025), which favors subsets with faster training-loss descent, unintentionally increases epiplexity and improves OOD perplexity and downstream tasks.

Emergence, Formalized

Definition 14: a dynamical system is 'epiplexity-emergent' if two compute limits T1 << T2 yield similar epiplexity for one-step prediction but significantly higher epiplexity for the low-compute observer on multi-step prediction. Low-compute observers cannot brute-force k-step simulation and must learn shortcuts—emergent patterns (gliders, oscillators) and their collision rules. Experiments with ECA Rule 54 show looped transformers exceeding a compute threshold can brute-force simulation, and epiplexity *drops*—a reversed form of emergence.

Implications for AI

1. Data selection theory: maximize epiplexity rather than relying on 'diversity/quality' heuristics. 2. Synthetic data legitimacy: DPI doesn't forbid information gain for bounded observers when the synthesis process is hard to invert. 3. Why LLM pretraining works: text has the highest epiplexity density. 4. Curriculum learning: reversed order, masked inputs, and inference-demanding tasks raise epiplexity. 5. Emergence is not magic: it is the necessary consequence of low-compute observers learning high-complexity shortcuts.

Key Numbers

• arXiv 2601.03220, v1 2026-01-06, v2 2026-03-16; CMU + NYU
• Code: https://github.com/shikaiqiu/epiplexity
• Core definition: Definition 8 (epiplexity S_T and time-bounded entropy H_T)
• ECA Rule 54: high epiplexity + moderate entropy; Rule 30: maximal entropy + zero epiplexity
• Natural-data epiplexity: text > chess > images; VQ tokenization greatly improves image epiplexity
• Loop transformers: epiplexity rises below a compute threshold, drops sharply above it

Conclusion

Epiplexity is not a replacement for Shannon entropy but its computationally bounded counterpart—removing the 'god's-eye' observer from information theory and replacing it with one that works within a 6ND FLOP budget. In this frame, seemingly contradictory phenomena become coherent: AlphaZero does learn new information (inverse policy search is intractable), synthetic data does help, data ordering matters, emergence is a computational inevitability, and text is special for its structural density. As usable data runs out, reframing data value from 'how many bits' to 'how much learnable structure' may matter more than any architectural improvement.