Directional Motion Blindness: Why Video-LLMs That See Everything Can't Tell Left from Right
> "The first principle is that you must not fool yourself — and you are the easiest person to fool." — Richard Feynman, 1974
The Phenomenon
Researchers Jongseo Lee, Hyuntak Lee, and Sunghun Kim found a striking failure in state-of-the-art video large language models (Video-LLMs). When shown the simplest possible videos—a single object moving in a straight line (left, right, up, or down) against a plain background—most models performed near chance level (~25%) at identifying the motion direction. The authors name this phenomenon Directional Motion Blindness.
Apparent above-chance performance turned out to be largely prediction bias (e.g., always guessing "right" when "right" happens to be frequent in the test set), not genuine directional understanding.
Diagnosing the Pipeline
The authors traced direction information through the standard Video-LLM architecture (vision encoder → projector → LLM):
- Vision encoder output: direction is linearly separable — the information exists.
- Projector output: direction remains linearly separable — nothing is lost in translation.
- LLM hidden states: direction remains linearly separable, though its separability margin gradually decays in deeper layers.
- MoDirect-SynBench: synthetic videos for in-domain testing
- MoDirect-RealBench: real-world videos for generalization testing
- No language labels needed — the model learns a low-level, general motion representation rather than memorizing verbal answers
- Works at the projector level — direction is explicitly extracted before reaching the LLM, without disturbing the pretrained vision encoder or adding costly tasks to the LLM
- Uses frame deltas — motion is fundamentally change, and change is most directly captured by feature differences
- Basic perception gaps: Similar deficits may exist in depth perception, temporal ordering, and causal reasoning in multimodal AI. The remedy is minimal-capability testing: test simple behaviors, not complex ones.
- Explicit intermediate representations: Explicitly predicting physical quantities (motion vectors, depth, optical flow) at intermediate layers may foster genuine understanding rather than surface correlation.
- From description to understanding: Evaluation should probe whether models grasp physical processes, not just whether their descriptions are fluent.
- Cross-modal binding: The Direction Binding Gap raises fundamental questions about how perception and language labels are bound together in neural networks.
- Lee, J., Lee, H., & Kim, S. (2025). *Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs*. arXiv preprint, cs.CV.
- Zellers, R., et al. (2019). From Recognition to Cognition: Visual Commonsense Reasoning. CVPR 2019.
- Li, K., et al. (2023). VideoChat: Chat-Centric Video Understanding. arXiv preprint.
- Maaz, M., et al. (2023). Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. arXiv preprint.
- Lin, K. Q., et al. (2023). UniAD: A Universal Model for Action Detection. CVPR 2023.
The Direction Binding Gap
The key finding: direction information is present at every stage, but models cannot bind this perception to the correct linguistic answer ("left", "right", "up", "down"). The model perceives direction but was never properly taught to map that perception to language labels.
Why Synthetic Data Fails
Instruction tuning on synthetic videos improves accuracy on synthetic benchmarks, but concept vector analysis reveals why this is superficial: visual complexity weakens the directional concept-vector signal. In simple videos the "left" signal is loud and clear; in real footage (CCTV, films, YouTube) it is drowned in noise.
The authors built two benchmarks:
After synthetic fine-tuning, models still perform poorly on real videos—only a few points above baseline. Synthetic training cannot fix directional blindness in the real world.
The Fix: DeltaDirect
DeltaDirect is a diagnostic-driven objective that trains the model to regress a normalized 2D motion vector from the feature difference between adjacent frames:
1. Take projected features of adjacent frames f₁ and f₂ 2. Compute Δf = f₂ - f₁ 3. Use a lightweight prediction head to predict (dx, dy) from Δf 4. Optimize with MSE loss against ground-truth motion
Key design choices:
Results
| Setup | MoDirect-SynBench accuracy | |---|---| | Baseline Video-LLM | 25.9% (near chance) | | DeltaDirect | 85.4% |
On MoDirect-RealBench, DeltaDirect improved real-video directional accuracy by 21.9 percentage points with zero real-world training data, while preserving standard video understanding performance. This indicates DeltaDirect learns a general capability grounded in physical regularities, not statistical memorization.