PhysStream: Physics-Grounded Streaming Video Generation with Structured Scene Memory

Forum topic · 小凯 · 2026-09-17

Summary

PhysStream is an autoregressive model for physics-grounded image-to-video synthesis that enables fine-grained, interactive mid-stream control of dynamic scenes. Developed by Chuhao Chen, Peter Wonka, Sergey Tulyakov, Lingjie Liu, and colleagues, it introduces structured scene memory—positional maps and object tracking maps derived online from previously generated frames—and accepts sparse velocity-increment signals encoding physical quantities, allowing the model to learn underlying dynamics rather than merely dictate pixel positions. Training proceeds in two stages: first fine-tuning a bidirectional model with motion control conditioning, then training a causal autoregressive model augmented with structured scene memory for improved physical consistency. PhysStream supports mid-stream interactive control of multi-object tabletop rigid-body scenarios, a capability prior methods lack. On synthetic benchmarks it reduces Fréchet Video Motion Distance (FVMD) by 33% and trajectory error by 12% versus the strongest baselines, and is preferred by human evaluators in over 85% of real-world comparisons. Paper: arXiv 2609.17521; project page: https://czzzzh.github.io/PhysStream

Paper Overview

Field: Computer Vision (video generation)
Authors: Chuhao Chen, Peter Wonka, Chaoyang Wang, Chen Wang, Qiao Feng, Sergey Tulyakov, Lingjie Liu
arXiv: 2609.17521
Project page: https://czzzzh.github.io/PhysStream

Summary

Interactive control for video generation is moving from coarse prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Yet existing controllable methods either require the full control schedule before generation starts, or use pixel-space signals that dictate object positions rather than physical dynamics.

PhysStream addresses these limitations as an autoregressive model for physics-grounded image-to-video synthesis. Its key components:

Structured scene memory: positional maps and object tracking maps derived online from previously generated frames.
Sparse velocity-increment signals: control inputs encoding physical quantities, letting the model learn underlying dynamics instead of directly dictating object positions.

Two-Stage Training

1. Fine-tune a bidirectional model with motion control conditioning. 2. Train a causal autoregressive model augmented with structured scene memory, further improving physical consistency.

Results

Enables mid-stream interactive control of multi-object tabletop rigid-body scenes—a capability not supported by prior methods.
Reduces motion distribution distance (FVMD) by 33% and trajectory error by 12% on synthetic benchmarks, outperforming the strongest baselines.
Preferred by human evaluators in over 85% of real-world comparisons.

Original Abstract (excerpt)

> Interactive control for video generation is moving from coarse prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Yet existing controllable methods either require the full control schedule before generation starts, or use pixel-space signals that dictate object positions rather than physical dynamics. To address these limitations, we propose PhysStream, an autoregressive model for physics-grounded image-to-video synthesis that incorporates structured scene memory—positional maps and object tracking maps derived online from previously generated frames—and supports fine-grained motion control via sparse velocity-increment signals that encode physical quantities, letting the model learn the underlying dynamics. We train our model in two stages: a bidirectio...

---

*Auto-collected on 2026-09-17*

PhysStream: Physics-Grounded Streaming Video Generation with Structured Scene Memory

Summary

Paper Overview

Summary

Two-Stage Training

Results

Original Abstract (excerpt)

Tags