Playing Videos Backwards to AI: Do Video Generation Models Understand Causality or Just Memorize the Arrow of Time?

Have you ever tried playing a video backwards?

A shattered glass reassembling, flames reborn from smoke, ink flowing back into a pen — these scenes look absurd because they violate our basic intuition about causality. Even a five-month-old infant shows "surprise" when watching a reversed video.

So what about the video generation AIs that claim to become "world models"? Do they get "surprised" when shown a reversed video?

A team from National Yang Ming Chiao Tung University and Shanghai AI Research (Tokyo) answered this question in a paper called YoCausal. The answer is unsettling: most video generation models can perceive temporal direction, but do not truly understand causality.

Inspiration from an Infant Cognition Lab

The story begins in 1987, when cognitive scientists Leslie and Keeble ran a classic experiment: showing infants normally played and reversed videos and observing their reactions. If infants showed "surprise" (longer looking time) at reversed videos, it indicated they had formed causal cognition — they "knew" that a hammer breaking a vase is a one-way event.

YoCausal's core insight is extremely elegant: apply this paradigm directly to generative models.

In the generative modeling framework, "surprise" has a precise mathematical counterpart — low probability. If a model truly understands causality, it should assign higher probability to normal videos and lower probability to reversed ones. Since a diffusion model's denoising loss is an upper bound on negative log-likelihood, the "degree of surprise" can be directly quantified via denoising loss.

No synthetic data needed, no lab recordings, no manually annotated counterfactual samples. Any real video, when reversed, becomes a natural counterfactual control. Zero cost, infinitely scalable.

Two-Level Testing: Time Sense ≠ Causal Sense

But there's a trap here.

Reversing a video introduces two anomalies simultaneously: reversed temporal direction and reversed causal relations. A model being "surprised" at a reversed video might just be detecting time flowing backwards, not understanding causality.

Consider two videos: one of a hammer shattering a vase (causal), one of a car driving on a highway (non-causal). Reversing the hammer video inverts both causality and time; reversing the car video inverts only time — a car driving backwards is odd, but there's no "causal inversion."

If a model truly understands causality, it should be more surprised by the reversed hammer-and-vase than the reversed car. This difference is YoCausal's core metric.

Specifically, YoCausal uses a two-level evaluation:

Level 1: Reverse Surprise Index (RSI) — measures a model's ability to perceive the arrow of time. For each video, add the same noise to the forward and reversed versions and compare denoising losses. If the model's denoising loss is higher on the reversed version (more "surprised"), it "perceives" temporal direction. RSI is the model's accuracy rate.

Level 2: Causal Cognition Index (CCI) — a vision-language model (VLM) splits the dataset into "causal" and "non-causal" subsets, then CCI = RSI(causal) − RSI(non-causal). Higher CCI means greater extra sensitivity to causal inversion — genuine causal understanding rather than mere time perception.

An Exam for 13 Models

The team tested 13 open-source video diffusion models with YoCausal, including Wan2.1/2.2, CogVideoX, HunyuanVideo, LTX-Video, and AnimateDiff, spanning different architectures and parameter scales. Datasets cover four domains: daily life, physical phenomena, human actions, and animal behavior.

The results reveal several key findings:

Finding 1: Perceiving time direction ≠ understanding causality. This is the most important conclusion. LTX-Video-13B and HunyuanVideo rank high on RSI (time perception) but perform poorly on CCI (causal cognition). They can detect that "time is flowing backwards" but cannot distinguish that "causality is inverted." Like a person who can see a video is reversed but doesn't understand why reversed playback looks wrong.

Finding 2: The best models are still far from humans. Humans nearly max out RSI, and their CCI far exceeds all models. The best performer, Wan2.2-A14B, correctly captures causal relations like "stains gradually disappearing while wiping a plate," but the gap with humans remains significant.

Finding 3: Causal cognition has zero correlation with visual quality. The Kendall correlation coefficient between causal ranking and Aesthetic Quality is τ = 0.0000. A model can generate gorgeous videos while completely failing to understand causality. Beautiful does not mean smart.

Finding 4: Scaling laws still hold for causal cognition. Correlation between parameter count and causal ranking is r = 0.688; between release date and causal ranking, r = 0.596. Bigger, newer models do have stronger causal understanding. The evolution from UNet to DiT architectures also brought significant improvement in causal perception.

Finding 5: Causal cognition ≠ physical intuition. The correlation between causal ranking and the physical intuition benchmark LikePhys is τ = 0.5111 — moderate but not reducible. Knowing physics and understanding causality are different things — a model may know a ball will bounce, but not understand "I threw the ball, therefore it flew."

An Elegant "Cheat Detector"

What I appreciate most about YoCausal is its methodological simplicity.

Traditional physics benchmarks require carefully synthesized data: simulate a ball rolling down a slope, then make it violate gravity and roll upward, and see whether the model detects the anomaly. But synthetic data has a sim-to-real gap — good performance on synthetic scenes doesn't mean real-world competence.

YoCausal bypasses this entirely. It needs nothing synthetic — just play real videos backwards. The operation is so simple you wonder: why didn't anyone do this before?

The answer: previous work conflated "temporal direction" with "causal relations." YoCausal's contribution isn't just "reversing videos" — it's the two-level metrics that decouple time perception from causal cognition. This decoupling is the key.

Lessons for Engineers

For engineers working on video generation models, YoCausal offers several practical insights:

1. Don't be fooled by RSI. Your model may be highly sensitive to temporal direction (high RSI) while clueless about causality (low CCI). RSI alone gives false confidence.

2. Causal evaluation needs dedicated design. Causal cognition has zero correlation with visual quality and only moderate correlation with physical intuition. You cannot substitute existing quality metrics or physics benchmarks for causal evaluation.

3. Scale is a remedy, not a cure. Bigger models are indeed stronger, but even the best models fall far short of humans. Architectural evolution (UNet → DiT) may deliver more than simply stacking parameters.

4. This benchmark can expand indefinitely. Any new video can join the test set without extra annotation. As models improve, the benchmark can evolve in step, never going "stale."

Limitations and Future Work

YoCausal has its boundaries. Time-symmetric events (like a Newton's cradle) look identical forwards and backwards, so RSI is powerless there. Also, computing denoising loss requires access to model weights, so closed-source models cannot be evaluated directly — though developers can use it internally.

A deeper philosophical question remains: does measuring causal understanding through "surprise" truly capture the essence of causality? A model surprised by reversed videos might simply never have seen reversed videos in training data, rather than understanding causality. YoCausal partially answers this with CCI — if it were just unfamiliarity, the model should be equally surprised by all reversed videos, not more surprised at causal ones. But the argument isn't perfect.

Still, as the paper title suggests — "How Far is Video Generation from World Model?" — YoCausal provides a clear distance metric. The answer: still far. But at least now we know how to measure it.

---

Paper: YoCausal: How Far is Video Generation from World Model? A Causality Perspective

Code: github.com/youzhe0305/YoCausal

Project page: youzhexie.me/papers/YoCausal