OpenCoF Deep Dive: When AI Learns to Reason Through Video Generation

Forum topic · 小凯 · 2026-07-10

Summary

This Chinese-language forum post on zhichai.net is a detailed walkthrough of OpenCoF: Learning to Reason Through Video Generation (arXiv:2607.08763). The author contrasts Chain-of-Thought (CoT) reasoning in large language models with a proposed Chain-of-Frame (CoF) paradigm, in which reasoning unfolds natively across video frames rather than through text. Key contributions covered include: (1) OpenCoF-17K, a 17,000-video dataset spanning 11 reasoning task families such as physical, causal, counterfactual, temporal-ordering, and compositional reasoning, built via procedural generation, human annotation, semi-automatic synthesis, and language-to-video conversion; (2) Wan-CoF, a supervised fine-tune of Alibaba's Wan2.1-I2V-14B video generation model trained to produce physically and logically consistent continuations, video answers, and counterfactual videos; and (3) reported benchmark gains, including +15.3% accuracy on MME-CoF and roughly 16-point improvements on physical reasoning F1 versus the base model. The post also explains the paper's dual token system separating visual reasoning tokens from textual reasoning tokens, attention-analysis findings across model depth and denoising steps, and open challenges: compositional generalization, long-horizon reasoning beyond ~100 frames, and whether predictive generation constitutes genuine causal understanding.

OpenCoF: Learning to Reason Through Video Generation — Deep Dive (Translated Summary)

This post is an editorial deep-dive (in the Feynman-explainer style typical of zhichai.net) of the paper OpenCoF: Learning to Reason Through Video Generation (arXiv:2607.08763). Below is a structured English rendering of its main arguments and findings.

Key points

From words to frames. Chain-of-Thought (CoT) made LLMs express intermediate steps in text and dramatically improved reasoning benchmarks (e.g., GSM8K accuracy rising from ~18% to 57%). But the author argues that the world does not happen "in language": physical, spatial, and temporal causality is better captured visually. Chain-of-Frame (CoF) proposes reasoning as a chain of video frames, analogous to a film director showing causality through continuous shots rather than a novelist describing it.
Generation ≠ reasoning. Post-Sora video models can produce realistic motion via statistical pattern matching, but current video benchmarks mostly test *descriptive* tasks (video QA, captioning), not genuine inference tasks like counterfactual prediction ("what if the fridge were locked?"), temporal ordering, or physical prediction. Experiments cited in the paper suggest even frontier video models perform poorly on these.

OpenCoF-17K dataset

A 17,000-video dataset where each clip is designed as a reasoning problem, spanning 11 task families (physical, causal, counterfactual, temporal-ordering, compositional reasoning, etc.). Four parallel data pipelines:

1. Procedural generation with physics engines (e.g., MuJoCo) with annotated reasoning chains 2. Human annotation on real-world videos 3. Semi-automatic synthesis mixing real footage and generated elements 4. Language-to-video conversion turning text reasoning problems (e.g., GSM8K-style) into visual form

Wan-CoF model

Base model: Wan2.1-I2V-14B (Alibaba, 14B parameters), fine-tuned via supervised training on OpenCoF-17K.
Training objectives: video completion (logically/physically consistent continuation), video-form question answering (answering *in video*), and counterfactual video generation.
Core insight: reasoning can be learned through generation — forcing physically consistent, logically coherent outputs requires internal inference.

Reported results

MME-CoF: +15.3% accuracy over the Wan2.1 base model
Gen-ViRe: physical reasoning F1 from 0.42 → 0.61
VIPER: causal judgment accuracy 52.1% → 68.7%
RULER-Bench (100+ frame videos): +22.4% temporal reasoning accuracy over baseline

Dual Token System and interpretability

OpenCoF separates two token types:

Visual reasoning tokens: track spatial relations, motion, and physical interaction; attention concentrates on interaction regions (e.g., the space between colliding objects).
Textual reasoning tokens: encode abstract logic and causal rules; attention jumps discretely to key inference nodes (condition changes, conclusions).

Analysis findings: shallow layers are dominated by visual tokens, deeper layers by textual ones ("perceive first, reason later"); attention patterns shift across diffusion denoising steps, from broad exploration to focused confirmation.

Open challenges (as discussed in the post)

Compositional generalization: performance drops when tasks require combining unseen reasoning steps.
Long-horizon reasoning: accuracy on 100+ frame (~4s) videos is still only ~45%; humans track hours-long narratives effortlessly.
Depth of causal understanding: predicting what happens next may not equal understanding *why* — a philosophical question left open.

The author closes with a Bergson-inspired reflection: CoT operates on "spatialized" clock time, while CoF approaches lived *durée* — reasoning as a flowing stream of images rather than a chain of words.

References cited in the post

1. Chen, X., et al. (2026). OpenCoF: Learning to Reason Through Video Generation. arXiv:2607.08763. 2. Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. 3. Wang, X., et al. (2025). Wan2.1: A Comprehensive Video Generation Framework. Alibaba Group. 4. Bergson, H. (1907). Creative Evolution. 5. Lake, B. M., et al. (2017). Building Machines That Learn and Think Like People. Behavioral and Brain Sciences, 40, e253. 6. Yi, K., et al. (2020). CLEVRER: Collision Events for Video Representation and Reasoning. ICLR 2020. 7. Baradel, R., et al. (2020). Cophy: Counterfactual Learning of Physical Dynamics. ICLR 2020.