MoRight: Motion Control Done Right — A Director's Baton for Pixel-Level Causal Motion

*English adaptation of a Chinese forum post analyzing the MoRight paper on controllable video generation.*

The Problem: Entangled Strings on a Marionette

The author opens with a puppeteer metaphor: current AI video generation is like a puppet whose strings are tangled. Existing methods typically use a single signal (optical flow or motion trajectories) that conflates two fundamentally independent dimensions:

• Actor motion — what objects in the scene do
• Camera motion — how the viewpoint moves

Result: asking for a car to turn left may instead rotate the entire street; camera zoom-in becomes all objects magically growing.

A second problem is missing motion causality. Models learn kinematic displacement (objects can move from A to B) but not why things move or what moving causes — leading to ghosts walking through walls and objects passing through each other.

MoRight's Core Ideas

1. Decoupled motion control

MoRight factors motion into two orthogonal subspaces:

• Object motion in a canonical static view: users specify trajectories in a fixed reference frame, like a motion-capture stage, free of camera concerns.
• Canonical-to-target-view mapping: a Temporal Cross-View Attention mechanism learns how motion observed from the canonical view projects to any target camera view, tracking time-dependent correspondences as objects move.

This works because 3D motion is inherently decomposable into world-space object motion plus a projection from world coordinates to camera coordinates.

2. Active vs. passive motion (causality)

Every object's motion is split into:

• Active motion: intentional, user-specified movement (e.g., a ball rolls forward).
• Passive motion: reactive movement inferred from physical causality (e.g., a bottle topples when struck).

Rather than hard-coded physics rules, MoRight learns causal relations from large-scale video data: training data is decomposed into driven motions and reactive chains, then the model jointly learns canonical-view motion generation, cross-view projection, and passive-response prediction.

The framework supports both:

• Forward reasoning: given an active motion, predict consequences (car hits wall → cracks, deformation, debris).
• Inverse reasoning: given a desired passive outcome (all bottles fall), infer the required active motion.

Technical Architecture Highlights

• Temporal Cross-View Attention: standard attention Attention(Q, K, V) = softmax(QK^T / √d) V, but Queries come from canonical-view features and Keys/Values from the target view, with temporally dynamic correspondences.
• Causal motion modeling network: multi-scale (multi-step) causal reasoning, scene-graph representations (nodes = objects, edges = relations), and implicitly learned physical priors (inertia, collision elasticity, gravity).
• Multi-task loss: reconstruction loss, motion consistency across views (round-trip view transforms should recover the original motion), causal consistency, and a disentanglement loss ensuring camera changes don't perturb object representations in latent space.

Evaluation

• Synthetic benchmark (known camera/object ground truth): state-of-the-art on both camera and object motion estimation.
• Real-world videos: human raters judged outputs more physically plausible and controllable, especially in multi-object interaction scenes.
• Interaction-aware benchmark: outperformed baselines on both predicting action consequences and inferring required actions.

Ablations confirm all three pillars are necessary: removing cross-view attention breaks view-consistent motion; removing causal modeling causes objects to interpenetrate; removing the disentanglement loss re-couples camera and object control.

Applications and Broader Significance

• Virtual production (pre-visualizing camera and actor choreography)
• Game development with physically plausible environment animation
• Robotic simulation data generation
• VR/AR where virtual objects must interact realistically with the real world

The author frames MoRight as a step from correlation learning toward causal reasoning in AI — akin to a child maturing from noticing co-occurrence to understanding cause — and as a building block for physical digital twins. It also sketches a new creative paradigm: users keep control of key creative decisions (motion, camera) while AI handles physical and causal consistency.

References Cited in the Post

1. Liu, S., Ren, X., Shen, T., et al. "MoRight: Motion Control Done Right." arXiv preprint, 2026. 2. Ho, J., Jain, A., Abbeel, P. "Denoising Diffusion Probabilistic Models." NeurIPS, 2020. 3. Vaswani, A., et al. "Attention is All You Need." NeurIPS, 2017. 4. Pearl, J. "Causality: Models, Reasoning, and Inference." Cambridge University Press, 2009. 5. Battaglia, P.W., et al. "Relational inductive biases, deep learning, and graph networks." arXiv:1806.01261, 2018.