The Predictive-Causal Gap: Impossibility Theorem Shows SSL Representations Fail at Causality

Forum topic · 小凯 · 2026-05-07

Summary

A paper by Kejun Liu of Soochow University (arXiv:2605.05029) argues that self-supervised next-token prediction, the backbone of LLMs and world models, structurally excludes causal understanding. The paper proves an impossibility theorem: for any model class with linear encoders, there exist stable dynamical systems for which all predictive-risk minimizers strictly mismatch the system's causal subspace. When background environmental noise changes slowly relative to the target system, loss functions incentivize encoders to fit the environment instead of the system. Large-scale experiments across 2,695 Gaussian-dynamics neural network configurations showed average causal fidelity of only 0.49; raising feature dimension to N=100 cut prediction error by 92% while causal fidelity collapsed to 10^-8, indicating scaling deepens non-causal local optima. Operational grounding (restricting loss to system observables) only partially mitigates the gap, exposing fragility of JEPA-style world models under out-of-distribution tests. The authors conclude future foundation models need explicit causal intervention mechanisms built into loss design.

The Predictive-Causal Gap: An Impossibility Theorem and Large-Scale Neural Evidence

A recent paper by Kejun Liu (Soochow University), posted to arXiv as arXiv:2605.05029 (May 2026), challenges the prevailing assumption that self-supervised learning—particularly next-token prediction in LLMs and world models—naturally gives rise to an understanding of underlying causal structure. Using rigorous mathematical proof plus large-scale empirical evaluation, the paper identifies a fundamental structural flaw: optimal predictive representations systematically exclude optimal causal representations.

1. Theoretical foundation: the impossibility theorem

A common industry assumption holds that as predictive risk decreases, models automatically "emerge" an understanding of the causal laws of physical systems. The paper disproves this with an impossibility theorem:

> For any model class containing linear encoders, there exist stable dynamical systems such that all predictive-risk minimizers necessarily mismatch the system's causal subspace.

Core mechanism: when background environmental noise evolves slowly or remains stable relative to the target system, the loss function naturally incentivizes the encoder to capture the *environment* rather than the *system*, since predicting the environment yields lower error. The model becomes an extreme "environment fitter" rather than a "causal parser."

2. Dimensional collapse of representations

To validate the theorem, the team evaluated 2,695 neural network configurations based on Gaussian dynamics, revealing catastrophic representational drift:

Causal fidelity — the core metric measuring encoder sensitivity to the system's degrees of freedom — averaged only 0.49.
Scaling backfires: increasing feature dimension to N=100 reduced prediction error by 92%, but causal fidelity collapsed to 10^-8.

This demonstrates that scale alone cannot fix missing causality; instead, it locks models into non-generalizable, locally non-causal optima.

3. Limits of operational grounding

The study also examined operational grounding — restricting the prediction loss strictly to the system's observable variables to strip out environmental factors.

Results show this partially mitigates causal deviation. However, if the boundary between system and environment is not mathematically well-defined, the model still cannot fully recover the true causal dynamics. This exposes the root fragility of current generative world models (e.g., JEPA architectures) under out-of-distribution (OOD) testing.

Conclusion

The research draws a clear theoretical boundary for large model development: improving prediction no longer equals improving intelligence. Future foundation model architectures must introduce explicit causal intervention mechanisms at the level of loss function design. The paper suggests industry should stop blindly scaling against single predictive metrics and move toward neuro-symbolic systems with built-in causal structure.

---

Paper information

Title: The Predictive-Causal Gap: An Impossibility Theorem and Large-Scale Neural Evidence
Author: Kejun Liu
Affiliation: Soochow University
arXiv ID: 2605.05029
Published: 2026-05-06
Categories: cs.LG, cs.AI