Playing Videos Backwards to AI: Do Video Generation Models Understand Causality or Just Memorize Time's Arrow?

Have you ever played a video backwards?

A shattered glass reassembling into a whole, flames reborn from smoke, ink flowing back from water into a pen — these scenes look absurd because they violate our basic intuition about causality. Even a five-month-old infant shows "surprise" when watching a reversed video.

So what about video generation AIs that claim to become "world models"? Would they be "surprised" by a reversed video?

A team from National Yang Ming Chiao Tung University and Shengda AI Research (Tokyo) answered this question in a paper called YoCausal. The answer is unsettling: most video generation models can sense temporal direction, but do not truly understand causality.

An Idea Borrowed from an Infant Cognition Lab

The story begins in 1987, when cognitive scientists Leslie and Keeble ran a classic experiment: showing infants normally played and reversed videos and observing their reactions. If infants looked longer at reversed videos — a sign of "surprise" — it meant they had formed causal cognition: they "knew" that a hammer breaking a vase is a one-way process.

YoCausal's core insight is elegantly simple: apply this paradigm directly to generative models.

In the generative modeling framework, "surprise" has a precise mathematical counterpart — low probability. If a model truly understands causality, it should assign higher probability to normal videos and lower probability to reversed ones. Since the denoising loss of a diffusion model is an upper bound on negative log-likelihood, the degree of "surprise" can be directly quantified via the denoising loss.

No synthetic data, no lab filming, no manually annotated counterfactual samples needed. Any real video, played backwards, becomes a natural counterfactual control. Zero cost, infinitely scalable.

Two Levels of Testing: Temporal Sense ≠ Causal Sense

But there's a trap here.

Reversing a video introduces two kinds of anomaly at once: reversed temporal direction and reversed causal relations. A model's "surprise" at a reversed video might simply reflect its detection of time flowing backwards, not an understanding of causality.

Consider two videos: one of a hammer shattering a vase (causal), and one of a car driving on a highway (non-causal). Reversing the hammer video inverts both causality and time; reversing the car video only inverts time — a car driving backwards is odd, but no "causal reversal" is involved.

If a model truly understands causality, it should be more surprised by the reversed hammer video than by the reversed car video. That difference is YoCausal's core metric.

Specifically, YoCausal uses a two-level evaluation:

Level 1: Reversal Surprise Index (RSI) — measures a model's ability to perceive the arrow of time. For each video, add identical noise to the forward and reversed versions and compare denoising losses. If the model shows a higher denoising loss (more "surprise") on the reversed version, it "perceives" temporal direction. RSI is the proportion of videos where the model "answers correctly."

Level 2: Causal Cognition Index (CCI) — a vision-language model (VLM) splits the dataset into "causal" and "non-causal" subsets, and CCI = RSI(causal) − RSI(non-causal). A higher CCI indicates stronger sensitivity to causal reversal specifically — genuine causal understanding, not just temporal-direction detection.

A Major Exam for 13 Models

The team tested 13 open-source video diffusion models with YoCausal, including Wan2.1/2.2, CogVideoX, HunyuanVideo, LTX-Video, and AnimateDiff, spanning different architectures and parameter scales. The datasets cover four domains: daily life, physical phenomena, human actions, and animal behavior.

The results reveal several key findings:

Finding 1: Perceiving temporal direction ≠ understanding causality. This is the most important conclusion. LTX-Video-13B and HunyuanVideo rank high on RSI (temporal perception) but perform poorly on CCI (causal cognition). They can detect that "time is flowing backwards," but cannot distinguish that "causality is flowing backwards" — like someone who can tell a video is reversed but doesn't understand why it looks wrong.

Finding 2: Even the best models remain far from humans. Humans score nearly perfect on RSI, and their CCI far exceeds all models. The best performer, Wan2.2-A14B, correctly captures causal relations like "stains gradually disappearing while wiping a plate," yet the gap to humans remains significant.

Finding 3: Causal cognition has zero correlation with visual quality. The Kendall correlation between causal ranking and Aesthetic Quality is τ = 0.0000. A model can generate gorgeous videos while completely failing to understand causality. Beautiful does not mean smart.

Finding 4: Scaling laws still hold for causal cognition. Parameter count correlates with causal ranking at r = 0.688, and release date at r = 0.596. Bigger, newer models do have stronger causal understanding. The architectural evolution from UNet to DiT also brought significant gains in causal perception.

Finding 5: Causal cognition ≠ physical intuition. Causal ranking correlates with the LikePhys physical-intuition benchmark at τ = 0.5111 — a moderate positive correlation but not reducible to it. Knowing physical laws and understanding causality are two different things: a model may know a ball will bounce, but not understand the causal link "I threw the ball, so it flew."

An Elegant "Cheating Detector"

What I admire most about YoCausal is its methodological simplicity.

Traditional physics benchmarks require carefully synthesized data: simulate a ball rolling down a slope, then make it roll upward in violation of gravity, and see whether the model detects the anomaly. But synthetic data suffers a sim-to-real gap — good performance on synthetic scenes doesn't transfer to the real world.

YoCausal bypasses this entirely. It requires synthesizing nothing; just play real videos backwards. The operation is so simple you might ask: why didn't anyone do this before?

The answer: previous work conflated "temporal direction" with "causal relations." YoCausal's contribution isn't just "reversing videos" — it's decoupling temporal perception from causal cognition via two-level metrics. That decoupling is the key.

Takeaways for Engineers

For engineers building video generation models, YoCausal offers several practical insights:

1. Don't be fooled by RSI. Your model may be sensitive to temporal direction (high RSI) while knowing nothing about causality (low CCI). RSI alone gives false confidence.

2. Evaluating causality requires dedicated design. Causal cognition has zero correlation with visual quality and only moderate correlation with physical intuition. Existing quality metrics or physics benchmarks cannot substitute for causal evaluation.

3. Scale is a remedy, but not a cure-all. Bigger models are indeed stronger, but even the strongest models fall far short of humans. Architectural evolution (UNet → DiT) may yield more than simply stacking parameters.

4. The benchmark scales indefinitely. Any new video can join the test set without extra annotation. As models improve, the benchmark evolves with them — it won't go "stale."

Limitations and Future Directions

YoCausal has its own boundaries. Time-symmetric events (like a Newton's cradle) look identical forwards and backwards, so RSI is powerless there. Also, computing denoising loss requires access to model weights, so closed-source models cannot be directly evaluated — though developers can use it internally.

The deeper philosophical question: does measuring causal understanding through "surprise" really capture the essence of causality? A model's surprise at reversed videos might simply reflect never having seen reversed videos during training, rather than causal understanding. YoCausal partially addresses this with CCI — if the model merely lacked exposure to reversed videos, it should be equally surprised by all reversals, not more so on causal videos. But the argument isn't airtight.

Still, as the paper's title suggests — "How Far is Video Generation from World Model?" — YoCausal provides a clear measure of that distance. The answer: still far. But at least now we know how to measure it.

---

Paper: YoCausal: How Far is Video Generation from World Model? A Causality Perspective

Code: github.com/youzhe0305/YoCausal

Project page: youzhexie.me/papers/YoCausal