Paper Overview
Field: Computer Vision (CV) Authors: Chuhao Chen, Peter Wonka, Chaoyang Wang, Chen Wang, Qiao Feng, Sergey Tulyakov, Lingjie Liu arXiv: 2609.17521
Summary
Interactive control for video generation is moving from coarse prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Yet existing controllable methods either require the full control schedule before generation starts, or use pixel-space signals that dictate object positions rather than physical dynamics.
To address these limitations, the authors propose PhysStream, an autoregressive model for physics-grounded image-to-video synthesis that features:
- Structured scene memory: positional maps and object tracking maps derived online from previously generated frames.
- Fine-grained motion control via sparse velocity-increment signals that encode physical quantities, letting the model learn the underlying dynamics.
- Enables mid-stream interactive control of multi-object tabletop rigid-body scenarios — a capability not supported by prior methods.
- Reduces motion distribution distance (FVMD) by 33% on synthetic benchmarks.
- Reduces trajectory error by 12% (both improvements over the strongest baseline).
- Preferred by human evaluators in over 85% of real-world comparisons.
- arXiv: https://arxiv.org/abs/2609.17521
- Project page: https://czzzzh.github.io/PhysStream
The model is trained in two stages:
1. Fine-tuning a bidirectional model with motion-control conditioning. 2. Training a causal autoregressive model augmented with structured scene memory to further improve physical consistency.
Results
Links
*Auto-collected on 2026-09-17.*