Paper Overview
- Field: Computer Vision (CV)
- Authors: Jongseo Lee, Hyuntak Lee, Sunghun Kim
- Published: 2025-05-23
- arXiv: 2505.17389
- MoDirect dataset family: for motion-direction instruction tuning and evaluation.
- DeltaDirect: a diagnostic-driven, projector-level objective that predicts a normalized 2D motion vector from feature differences of adjacent frames.
- On MoDirect-SynBench, instruction tuning with DeltaDirect improves motion direction accuracy from 25.9% to 85.4%.
- On MoDirect-RealBench, DeltaDirect improves real-world motion direction accuracy by 21.9 percentage points without any real-world fine-tuning data, while maintaining standard video understanding performance.
Summary
Video large language models (Video-LLMs) have advanced rapidly in temporal video understanding, yet many fail at a fundamental perceptual primitive: signed image-plane motion direction. In simple videos where a single object moves left, right, up, or down, most Video-LLMs perform near chance level, and the few cases exceeding chance are largely attributable to prediction bias rather than genuine direction understanding. The authors term this failure directional motion blindness.
By tracing motion direction information through the Video-LLM pipeline, the paper finds that motion direction is linearly decodable in the vision encoder, the projector, and the LLM hidden states — but the readout mechanism fails to bind this signal to the correct linguistic answer option, exposing a direction binding gap.
While synthetic motion-direction instruction tuning reduces this gap on the source domain, motion direction concept vector analysis shows that visual complexity attenuates signal magnitude, limiting cross-domain generalization.
Contributions
Results
*Auto-collected on 2026-05-23*