Directional Motion Blindness: Video-LLMs Can't Tell Left From Right

Forum topic · 小凯 · 2026-05-22

Summary

A May 2026 paper from Kyung Hee University (arXiv:2605.22823, "Which Way Did It Move?") reveals a striking blind spot in video large language models (Video-LLMs): when shown simple videos of a dot moving left, right, up, or down, mainstream models — including GPT-4o, Gemini 2.5 Pro, and Qwen2.5-VL — score roughly 25% accuracy, equivalent to random guessing. Probing experiments show the direction information is present and linearly decodable throughout the visual encoder, projection layer, and LLM hidden states, but models fail to bind this signal to the correct textual answer — a phenomenon the authors call the Direction Binding Gap. Signal strength also degrades as videos become more visually complex. The proposed fix, DeltaDirect, adds an auxiliary training objective at the projection layer that predicts normalized 2D motion vectors from frame-to-frame feature differences. It lifts accuracy on synthetic benchmarks from 25.9% to 85.4% and improves real-video performance by 21.9 percentage points zero-shot, without degrading standard video understanding benchmarks. This matters for autonomous driving, surveillance, sports analytics, and AR interaction, where motion direction is a prerequisite for prediction.

Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs

| Field | Details | |---|---| | Authors | Jongseo Lee, Hyuntak Lee, Sunghun Kim, Sooa Kim, Jihoon Chung, Jinwoo Choi | | Institution | Kyung Hee University | | arXiv ID | 2605.22823 | | Date | May 21, 2026 | | Category | cs.CV | | Headline result | Video-LLMs are near random chance at detecting motion direction; DeltaDirect raises accuracy from 25.9% to 85.4% |

The surprising finding

Show a Video-LLM a three-second clip of a red dot moving from left to right on a plain white background and ask which way it moved. Most state-of-the-art models — capable of recognizing thousands of objects and understanding complex narratives — answer essentially at random. The authors built MoDirect-SynBench, a benchmark of trivially simple videos (single dot or shape, four possible motion directions, no occlusion or clutter), and tested mainstream models:

| Model | Direction accuracy | |---|:---:| | Random guess | 25.0% | | GPT-4o | ~27% | | Gemini 2.5 Pro | ~28% | | Qwen2.5-VL | ~26% | | LLaVA-Video | ~24% | | VideoLLaMA-2 | ~23% |

The small gains above chance are largely attributable to prediction bias (models favoring a particular answer option), not genuine motion understanding.

Diagnosis: the Direction Binding Gap

The authors probed each stage of the pipeline — video frames → visual encoder → projection layer → LLM → answer text — by training simple linear classifiers on hidden states to decode motion direction:

Visual encoder states: direction is clearly decodable.
After the projection layer: direction is still decodable.
Across LLM layers: direction remains linearly readable.

The signal never disappears — the model "knows" the direction but cannot bind it to the correct language output. The authors name this the Direction Binding Gap. They further show, via concept-vector analysis, that the direction vector weakens as video complexity increases: clean, single-object videos yield strong signals, while textured, multi-object scenes bury direction in noise. This also explains why the flaw went largely unreported — standard benchmarks ask about high-level semantics, which are often invariant to direction.

The fix: DeltaDirect

DeltaDirect adds an auxiliary objective at the projection layer, before information enters the LLM:

1. Compute the feature difference (delta) between adjacent frames from the visual encoder. 2. Predict the normalized 2D motion vector (horizontal and vertical components) from this delta. 3. Train the projection layer with this direction-prediction goal alongside its usual task.

The LLM itself is untouched — direction is simply upgraded from an implicit signal to an explicit marker before language processing begins.

Results:

MoDirect-SynBench (synthetic): accuracy rises from 25.9% to 85.4%.
MoDirect-RealBench (real videos): +21.9 percentage points, achieved zero-shot with no real-video training data.
Standard video understanding benchmarks: no performance degradation.

Why it matters

Direction is a prerequisite for action prediction. Applications affected include:

Autonomous driving: predicting where a cyclist will be next second requires knowing their direction of travel.
Surveillance: distinguishing "entering from the left" from "leaving to the right" is essential for intrusion judgments.
Sports analytics: misreading pass direction collapses tactical analysis.
AR/assistive interfaces: mistaking which direction a user is reaching causes wrong assistance.

Limitations acknowledged

Only four discretized directions were tested; continuous angles (e.g., 37.5 degrees up-right) were not.
Multi-object scenes with conflicting motions remain difficult even with DeltaDirect.
The causal link to downstream task improvements (e.g., better driving planning) is not yet validated.
The deep mechanism of why the binding gap exists — possibly related to the semantic overload of direction words in natural language pretraining — remains unexplained.

References

1. Lee, J., et al. (2026). Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs. *arXiv:2605.22823*. 2. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. *Journal of Physiology*. 3. Bai, J., et al. (2025). Qwen2.5-VL Technical Report. *arXiv:2502.13923*.

The paper echoes a classic result: Hubel and Wiesel's Nobel-winning discovery of direction-selective neurons in the cat visual cortex. Sixty years later, artificial visual systems with tens of billions of parameters still lacked this most basic capability — until a small patch wired the signal to the words "left" and "right."