The Predictive-Causal Gap: An Impossibility Theorem and Representation Collapse in Self-Supervised Learning
As AI advances toward general-purpose generalization, the self-supervised learning paradigm centered on "next-token prediction" (as in LLMs and world models) has been regarded as a universal underlying engine. However, a new paper by Kejun Liu of Soochow University, released in May 2026 as arXiv:2605.05029, breaks this consensus with rigorous mathematical derivation and large-scale empirical evidence. The research reveals a fundamental structural flaw: optimal predictive representations systematically exclude optimal causal representations.
1. Theoretical Foundation: The Impossibility Theorem
A widespread assumption holds that as predictive risk decreases, models will automatically "emerge" an understanding of the underlying causal laws of physical systems. The paper formalizes an impossibility theorem that falsifies this assumption.
The theorem proves that for any model class containing linear encoders, there exist stable dynamical systems such that all predictive-risk minimizers are strictly misaligned with the system's causal subspace.
> Core mechanism: When the dynamics of ambient background noise are slower than or stable relative to the target system, the loss function naturally incentivizes the encoder to capture the "environment" rather than the "system," because predicting the environment more easily achieves low error. The model thus becomes an extreme "environment fitter" rather than a "causal parser."
2. Dimensional Collapse of Representations
To validate the theorem, the research team conducted large-scale evaluations on 2,695 neural network configurations based on Gaussian dynamics, revealing catastrophic representational drift:
- Causal Fidelity: the core metric measuring encoder sensitivity to the system's degrees of freedom. Its mean value was only 0.49.
- The backlash of scaling laws: when feature dimension increased from low-dimensional settings to N=100, predictive error dropped significantly by 92%. Strikingly, however, causal fidelity collapsed to 10^-8.
- Title: The Predictive-Causal Gap: An Impossibility Theorem and Large-Scale Neural Evidence
- Author: Kejun Liu
- Institution: Soochow University
- arXiv ID: 2605.05029
- Published: 2026-05-06
- Categories: cs.LG, cs.AI
This demonstrates that scale expansion alone cannot resolve the absence of causality; instead, it locks models into non-generalizable local non-causal optima.
3. Limits of Operational Grounding
To salvage predictive learning, the study examines operational grounding—restricting the prediction loss strictly to "system-observable variables" to strip out environmental factors.
Experiments show this approach can partially alleviate causal misalignment. However, if the boundary between system and environment is not mathematically well-defined, the model still cannot fully recover the true causal dynamics. This exposes the root of fragility in current generative world models (e.g., JEPA-style architectures) when facing out-of-distribution (OOD) testing.
Conclusion
The research draws a clear theoretical boundary for foundation model development: improvement in predictive capability no longer equals progress toward intelligence. Future foundation model architectures must introduce explicit causal intervention mechanisms at the level of loss function design. For industry, abandoning blind scale worship of single predictive metrics in favor of building neuro-symbolic systems with causal structure is increasingly urgent.
---