MotiMotion: Motion-Controlled Video Generation with Visual Reasoning

Forum topic · 小凯 · 2026-05-25

Summary

This paper introduces MotiMotion, a new framework for image-to-video generation that reframes motion control as a 'reason-then-generate' process. Existing motion-controlled video models rigidly follow sparse, imprecise, and causally incomplete user-provided paths, often producing unnatural results that miss secondary causal effects. MotiMotion uses a training-free vision-language reasoner to refine image-space coordinates of the primary path and hallucinate plausible secondary motions grounded in causal, common-sense interactions. It also proposes a confidence-aware control scheme that modulates guidance strength, enabling the model to tightly follow high-confidence plans while using its internal generative prior to correct artifacts in low-confidence regions. The authors curate MotiBench, a new interactive image-to-video benchmark with motion-triggered events. Both automated VLM-based evaluations and human studies on MotiBench show that MotiMotion generates videos with more plausible object behaviors and interactions than prior methods.

Paper Overview

Research Area: Computer Vision (CV)
Authors: Lee Hsin-Ying, Hanwen Jiang, Yiqun Mei
Release Date: 2026-05-25
arXiv: 2505.14486

Key Points

Problem

Current motion-controlled image-to-video (I2V) models rigidly follow user-provided trajectories.
User paths are typically sparse, imprecise, and causally incomplete.
This often leads to unnatural or implausible results, especially missing secondary causal consequences of an action.

Method: MotiMotion

Reframes motion control as a reason-then-generate pipeline rather than direct trajectory following.
Uses a training-free vision-language reasoner to:
Refine the image-space coordinates of the primary (user-specified) motion path.
Hallucinate plausible secondary motions grounded in causal and common-sense interactions (e.g., when one object moves, neighboring objects should respond realistically).
Introduces a confidence-aware control scheme that modulates guidance strength:
High-confidence plan regions: the model tightly follows the refined trajectories.
Low-confidence input regions: the model falls back on its internal generative prior to suppress artifacts.

Benchmark: MotiBench

A newly curated image-to-video benchmark focused on interaction-centric scenes.
Scenarios are designed so that new events are triggered by motion, emphasizing causal chains and secondary effects.
Used for systematic comparison against existing methods.

Evaluation

VLM-based (Vision-Language Model) automated evaluation on MotiBench.
Human studies on MotiBench.
Both evaluations indicate that MotiMotion produces videos with:
More plausible object behaviors.
More coherent and physically grounded interactions.
Overall improvements over prior motion-controlled video generation approaches.

Significance

Demonstrates that injecting explicit visual reasoning before generation can address the causal incompleteness of raw user trajectories.
Shows that confidence-aware guidance balances user fidelity with generative realism.
Contributes a new benchmark (MotiBench) to drive future research in causally aware, interaction-rich video generation.

*Automatically collected on 2026-05-25*