Paper Overview
Research Area: CV Authors: Jongseo Lee, Hyuntak Lee, Sunghun Kim Published: 2026-05-25 arXiv: 2505.14483
Summary
Video large language models (Video-LLMs) have made rapid progress in temporal video understanding, yet many models fail on a basic perceptual primitive: signed image-plane motion direction. On simple videos where a single object moves left, right, up, or down, most Video-LLMs perform near chance level, with performance above chance largely attributable to prediction bias rather than genuine direction understanding. The authors call this failure directional motion blindness.
Key Findings
- Locating the failure: Tracing motion-direction information through the Video-LLM pipeline shows that direction remains linearly accessible in the visual encoder, projector, and LLM hidden states. The readout layers fail to bind this signal to the correct linguistic answer choices, revealing a direction-binding gap.
- Limits of naive tuning: Synthetic motion-direction instruction fine-tuning can close the gap in the source domain. However, motion-direction concept vector analysis shows that visual complexity attenuates signal magnitude and limits out-of-domain generalization.
- MoDirect: A dataset family for motion-direction instruction fine-tuning and evaluation.
- DeltaDirect: A diagnosis-driven, projector-level objective that predicts normalized 2D motion vectors from differences of adjacent-frame features.
- On MoDirect-SynBench, instruction tuning with DeltaDirect improves motion-direction accuracy from 25.9% to 85.4%.
- On MoDirect-RealBench, DeltaDirect improves real-world motion-direction accuracy by 21.9 percentage points over the vanilla baseline—without any real-world fine-tuning data—while maintaining standard video understanding performance.
- arXiv: 2505.14483