Information Is Not a Material Quantity: What Epiplexity Reveals About What AI Actually Extracts from Data

> Paper analyzed: arXiv 2601.03220 — *From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence* > Authors: Marc Finzi*, Shikai Qiu*, Yiding Jiang*, Pavel Izmailov, J. Zico Kolter, Andrew Gordon Wilson > Affiliations: Carnegie Mellon University, New York University > Published: 2026-01-06 (v2: 2026-03-16) > Code: https://github.com/shikaiqiu/epiplexity

Key points

• Classical information theory (Shannon entropy, Kolmogorov complexity) implicitly assumes observers with unlimited compute. The paper proposes epiplexity — the structured information a computationally bounded observer can extract from data.
• Three paradoxes resolved under bounded computation:
1. Deterministic transformations can't add information — yet AlphaZero learns superhuman policies from just the rules and self-play. Under bounded compute, outputs of hard-to-invert processes (CSPRNG, elementary cellular automata) contain genuinely new information for the observer. 2. Information should be order-invariant — yet text models strongly prefer left-to-right modeling. In chess, training on board→moves (reverse) yields higher epiplexity and better out-of-distribution downstream performance (centipawn evaluation) than moves→board. Theorem 13: assuming one-way permutations exist, forward/reverse time-bounded entropy can differ by ω(log n). 3. Likelihood training only matches distributions — yet models must learn programs more complex than the generating process to predict (e.g., inferring a masked ECA input by enumeration).
• Formal definition (Definition 8): fit the minimum time-bounded MDL two-part code; epiplexity S_T(X) is the program length (structure), time-bounded entropy H_T(X) is residual unpredictability. The structure/randomness boundary depends on the observer's compute budget.

Measurement methods

• Prequential coding: cumulative training loss (the area between the loss curve and final loss) estimates model description length. Cheap; slightly informal.
• Requential coding: teacher-student framework; coding cost ≈ cumulative KL(P^t_i ‖ P^s_i). Rigorous but 2–10× more expensive. Rankings from the two methods are highly consistent.

Empirical findings

• Elementary cellular automata: Rule 15 (Class II) — low epiplexity; Rule 30 (Class III, chaotic) — maximum entropy, near-zero epiplexity; Rule 54 (Class IV) — high epiplexity with moderate entropy.
• Natural data (5B tokens, ~6×10^18 FLOPs): epiplexity ranks text (OpenWebText) > chess > images (CIFAR-5M, where 99%+ of information is random pixel content). VQ tokenization substantially raises image epiplexity. Scaling-law extrapolation to 1T tokens / 10^25 FLOPs keeps language highest — explaining why text pretraining transfers best: it has the highest *structured-information density*, not the most total bits.
• Data selection: Adaptive Data Optimization (ADO, Jiang et al. 2025) implicitly maximizes epiplexity by favoring subsets with faster loss descent, improving OOD perplexity and downstream tasks.
• Emergence (Definition 14): a system is epiplexity-emergent if a low-compute observer must learn complex 'shortcut' programs (e.g., glider rules in Conway's Game of Life) to approximate multi-step evolution. Experiments with looped vs. non-looped transformers on Rule 54 show epiplexity rising below a compute threshold, then dropping sharply once brute-force simulation becomes feasible — a *reverse* emergence where more compute finds the simpler underlying rule.

Implications for AI

1. Data selection should aim to maximize epiplexity — learnable structure — rather than rely on heuristics. 2. Synthetic data can genuinely add information for bounded observers when the generation process is hard to invert (data-processing inequality does not bind). 3. Curriculum learning: data ordering, masked inputs, and inference-demanding tasks raise epiplexity even if they increase training loss. 4. Emergence is not magic: it is the necessary consequence of low-compute observers forced to learn high-complexity approximations of simple rules.

Assessment

The theory rests on standard cryptographic assumptions (existence of one-way functions) and is corroborated by neural-network experiments. The main point of contention may be the rigor of prequential 'area' estimates, mitigated by requential coding as a backup. Epiplexity is a task-agnostic measure — it quantifies how much reusable structure data contains, not whether that structure is useful for a given task — and complements entropy: entropy says how much unpredictability remains; epiplexity says how much structure has been internalized. The paper reframes 'the value of data' from raw bits to learnable structure — a critical perspective as high-quality data becomes scarce.

---

> Reference: Finzi et al., arXiv:2601.03220 [cs.LG] (2026). Code: https://github.com/shikaiqiu/epiplexity