Directional Motion Blindness: Video-LLMs Can Describe Complex Scenes but Fail to Tell Left from Right

Forum topic · 小凯 · 2026-05-22

Summary

Researchers Jongseo Lee, Hyuntak Lee, and Sunghun Kim identify "Directional Motion Blindness" in Video Large Language Models (Video-LLMs): on simple videos of a single object moving against a plain background, most state-of-the-art models perform near chance (about 25% for four directions). Through linear-probe analysis, the authors show motion direction remains linearly decodable across the vision encoder, projector, and LLM hidden states, yet models cannot bind this perceptual signal to the correct language label — a phenomenon called the Direction Binding Gap. Synthetic instruction tuning fixes performance on synthetic data but fails to generalize to real-world video, where visual complexity weakens the direction concept vectors. The proposed remedy, DeltaDirect, is a diagnostic-driven objective that trains a lightweight head at the projector level to predict a normalized 2D motion vector from feature differences between adjacent frames. DeltaDirect lifts accuracy on MoDirect-SynBench from 25.9% to 85.4% and improves real-video accuracy by 21.9 percentage points without real-world training data, while preserving standard video understanding. The work exposes foundational perception gaps in multimodal AI and argues for explicit physical-quantity supervision.

Directional Motion Blindness: AI Eyes That See Everything but Cannot Tell Left from Right

> "The first principle is that you must not fool yourself — and you are the easiest person to fool." — Richard Feynman, 1974

Source: Lee, J., Lee, H., & Kim, S. (2025). *Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs* (arXiv preprint, cs.CV). The following is a structured English rendering of a Chinese forum post analyzing this paper.

Key points

The phenomenon: Given trivially simple videos — a single geometric object moving in a straight line (left/right/up/down) against a plain white, black, or gray background — most state-of-the-art Video-LLMs perform near random chance (~25%). Apparent above-chance results are largely explained by prediction bias (e.g., always guessing "right").
The architecture in question: Video-LLMs typically chain a vision encoder (e.g., ViT), a projector mapping visual features into the LLM's input space, and an LLM that generates text answers.

Where does the information get lost? Information-flow tracing

The authors probe each stage with linear classifiers to test whether motion direction is linearly decodable:

1. Vision encoder output: direction is linearly separable. 2. Projector output: direction remains linearly separable — the projector does not distort or discard it. 3. LLM hidden states: direction remains linearly decodable through deep layers, though the classification margin gradually shrinks toward output layers.

The Direction Binding Gap: the information exists everywhere in the pipeline, but the model cannot *bind* it to the correct linguistic answer ("left", "right", etc.). The failure is not perception; it is never having been properly taught the perception-to-language mapping.

Why synthetic data alone fails

Instruction tuning on synthetic videos (MoDirect-SynBench) improves synthetic-domain accuracy substantially — but this is a surface fix.
Concept vector analysis shows visual complexity attenuates the magnitude of direction signals: in real footage (CCTV, movies, YouTube), the "leftward" concept vector becomes a whisper drowned in noise.
On the real-video benchmark MoDirect-RealBench, synthetically fine-tuned models still perform only slightly above baseline. The learned binding is a fragile association, not robust understanding.

The fix: DeltaDirect

DeltaDirect is a diagnostic-driven objective that predicts motion directly rather than memorizing answers:

1. Take projected features of adjacent frames, f1 and f2. 2. Compute the feature difference Δf = f2 − f1. 3. A lightweight prediction head maps Δf to a normalized 2D motion vector (dx, dy). 4. Training loss is mean squared error against the ground-truth motion vector.

Design rationale:

Language-label-free: the model learns a low-level, general motion representation instead of memorizing verbal labels.
Placed at the projector: this preserves the pre-trained vision encoder's generic representations and avoids expensive modifications to the LLM. The projector is the perfect intermediate layer connecting both modalities.
Delta-based: motion is fundamentally change, and change is most directly expressed in feature differences between adjacent frames.

Results

| Setting | Accuracy | |---|---| | Baseline Video-LLM (MoDirect-SynBench) | 25.9% (near chance) | | DeltaDirect (MoDirect-SynBench) | 85.4% | | DeltaDirect (MoDirect-RealBench, zero real training data) | +21.9 percentage points, with standard video understanding preserved |

The real-video gain without real-world training data suggests DeltaDirect learns a transferable *capability* grounded in physics (frame-difference → motion vector) rather than statistical memorization.

Deeper implications

High-level "understanding" may rest on low-level misunderstanding. Models that describe complex scenes fluently can fail at the simplest perceptual primitives, echoing Feynman's warning that "nature cannot be fooled."
Other fundamental gaps likely exist: depth perception, temporal ordering, and causal reasoning could harbor analogous binding gaps, diagnosable with the same minimalist-test methodology.
Methodological lesson: to test whether a complex system truly understands X, test its simplest behaviors, not its complex ones.
Future directions: systematic minimal-capability evaluations, explicit intermediate physical-quantity supervision (motion vectors, depth, optical flow), shifting evaluation from description quality to understanding depth, and studying cross-modal binding mechanisms in multimodal models.

References

Lee, J., Lee, H., & Kim, S. (2025). *Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs*. arXiv preprint. cs.CV.
Zellers, R., et al. (2019). From Recognition to Cognition: Visual Commonsense Reasoning. *CVPR 2019*.
Li, K., et al. (2023). VideoChat: Chat-Centric Video Understanding. arXiv preprint.
Maaz, M., et al. (2023). Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. arXiv preprint.
Lin, K. Q., et al. (2023). UniAD: A Universal Model for Action Detection. *CVPR 2023*.