Direction Blindness: Video-LLMs That See Everything Can't Tell Left from Right

Forum topic · 小凯 · 2026-05-22

Summary

A 2026 paper from Kyung Hee University (arXiv:2605.22823, 'Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs') reveals that leading video large language models (Video-LLMs) perform near chance (~25%) on a basic perception task: identifying whether a simple moving object travels left, right, up, or down. Models including GPT-4o, Gemini 2.5 Pro, Qwen2.5-VL, LLaVA-Video, and VideoLLaMA-2 all hover around random guessing on the MoDirect-SynBench benchmark. Probing experiments show the directional signal is linearly readable throughout the vision encoder, projection layer, and LLM — the failure is a 'Direction Binding Gap': the model knows the direction but cannot bind it to the correct verbal answer. Signal strength also degrades as videos become more complex. The proposed DeltaDirect method adds an auxiliary objective to the projection layer, predicting normalized 2D motion vectors from inter-frame feature deltas, raising accuracy from 25.9% to 85.4% on synthetic videos and +21.9 points on real videos zero-shot, without degrading standard benchmarks. The finding matters for autonomous driving, surveillance, sports analytics, and AR interaction, where motion direction underpins prediction and action.

| Paper | | |---|---| | Title | Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs | | Authors | Jongseo Lee, Hyuntak Lee, Sunghun Kim, Sooa Kim, Jihoon Chung, Jinwoo Choi | | Institution | Kyung Hee University | | arXiv | 2605.22823 | | Date | May 21, 2026 | | Code | Open-sourced |

Key points

Near-random directional perception: On MoDirect-SynBench — videos of a single dot or shape moving purely left, right, up, or down — mainstream Video-LLMs score ~25% (chance level for four directions).

| Model | Direction accuracy | |------|:---:| | Random guess | 25.0% | | GPT-4o | ~27% | | Gemini 2.5 Pro | ~28% | | Qwen2.5-VL | ~26% | | LLaVA-Video | ~24% | | VideoLLaMA-2 | ~23% |

The marginal gains above chance are attributed mainly to prediction bias (a tendency to pick certain labels), not genuine understanding.

Diagnosis: the signal is there, but it can't be spoken

A Video-LLM pipeline runs: video frames → vision encoder → projection layer → LLM → answer text. Linear probes at each stage show the direction signal is clearly decodable in the vision encoder, in the projection layer, and throughout the LLM's hidden states — the information never disappears. The authors name the failure the Direction Binding Gap: the model internally represents motion direction but cannot bind it to the correct linguistic output.

Concept-vector analysis shows the directional signal weakens as video complexity increases (added textures, backgrounds, multiple moving objects), lowering signal-to-noise. This also explains why the defect went unnoticed: standard benchmarks ask semantic questions ("what is happening?") whose answers don't depend on direction.

The fix: DeltaDirect

DeltaDirect adds an auxiliary training objective to the projection layer:

1. Compute feature deltas between adjacent frames (subtract encoder outputs of frames t and t+1). 2. Predict the normalized 2D motion vector (horizontal and vertical components) directly from the delta. 3. Train the projection layer with this prediction as a secondary objective; the LLM itself is unchanged.

Results:

MoDirect-SynBench (synthetic): accuracy rises from 25.9% → 85.4%.
MoDirect-RealBench (real videos): +21.9 points via pure zero-shot transfer, with no real-video training data.
Standard video benchmarks: no performance regression.

Why it matters

Direction is a prerequisite for action: autonomous driving must predict where a cyclist is heading; surveillance must distinguish entering from leaving; sports analytics must know where a pass is going; AR assistants must see which way a user reaches. Vision intelligence that cannot resolve direction is effectively blind in any predictive, interactive setting.

Limitations acknowledged

Only four discretized directions are tested; continuous-angle motion is not evaluated.
Multi-object scenes with objects moving in different directions remain difficult even with DeltaDirect.
The causal chain — whether fixing direction perception improves downstream tasks like driving — is unverified.
The deep mechanism of the binding failure remains unexplained; the authors speculate it relates to how rarely language pretraining requires inferring direction directly from visual signals, and to the semantic overload of "left/right" in language.

References

1. Lee, J., et al. (2026). Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs. *arXiv:2605.22823*. 2. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. *Journal of Physiology*. 3. Lin, J., et al. (2024). Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. *EMNLP 2024*. 4. Li, K., et al. (2024). LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data? *arXiv:2405.01542*. 5. Bai, J., et al. (2025). Qwen2.5-VL Technical Report. *arXiv:2502.13923*.