Directional Motion Blindness in Video-LLMs: When AI Loses Its Sense of Direction
> *"Seeing is not understanding. The image on the retina is only a dance of photons; real vision begins the moment the brain assigns direction to motion."* — adapted from Oliver Sacks, *The Man Who Mistook His Wife for a Hat*
This is an English rendering of a detailed Chinese forum post walking through the paper "Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs" by Jongseo Lee, Hyuntak Lee, and Sunghun Kim (KHU-VLL, Kyung Hee University).
The core finding
State-of-the-art Video-LLMs can summarize movies, answer complex video QA questions, and score well on multimodal benchmarks — yet when asked which direction an object moved (left/right/up/down), they perform at roughly 25%, i.e., random-guess level, even on trivial scenes with a solid background and a single object. The authors name this Directional Motion Blindness.
The post opens with an analogy: at a banquet, you notice every detail of a woman in a red dress who just entered — her velvet dress, pearl earrings, her ring — but cannot say whether she came in from the left or the right.
Where does the information get lost? A pipe-tracing experiment
The authors trace motion-direction information through the Video-LLM pipeline like a fish swimming through three pipe segments, testing each stage with linear probes:
- Vision encoder: linear probes easily decode motion direction from the encoder's features. Information is intact.
- Projector: direction remains linearly accessible after modality translation. Still intact.
- LLM hidden states: even here, direction is linearly recoverable from hidden states.
- Language output: yet when asked to emit "left"/"right"/"up"/"down", accuracy collapses to ~25%.
- MoDirect-SynBench: synthetic data for training basic direction ability (simple scenes, precise labels).
- MoDirect-RealBench: real-world data for evaluating generalization to complex scenes.
- Title: Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs
- Authors: Jongseo Lee, Hyuntak Lee, Sunghun Kim
- Affiliation: KHU-VLL, Kyung Hee University
- Category: cs.CV
- Code: https://github.com/KHU-VLL/DeltaDirect
The paradox: the information exists inside the model but cannot be bound to the correct language label. The authors call this the Direction Binding Gap — like recognizing a friend's face but being unable to recall their name.
Why naive fine-tuning fails
The first fix attempt — synthetic motion direction instruction tuning on simple videos (a colored square sliding on a gray background, labeled with directions) — improves in-domain (synthetic) accuracy but fails to transfer to real-world videos with textures, occlusion, and clutter.
Using Motion Direction Concept Vector Analysis, the authors show that increased visual complexity weakens the magnitude of the motion-direction signal in feature space and degrades its alignment with language labels — like a solo soprano drowned out by a full orchestra.
DeltaDirect: predict motion from frame differences
The breakthrough exploits an elegant insight: motion *is* the difference between adjacent frames. DeltaDirect is a diagnostic-driven objective applied at the projector level:
1. Extract features \(F_t\) and \(F_{t+1}\) for adjacent frames via the vision encoder. 2. Compute the feature delta \(\Delta F = F_{t+1} - F_t\) (in high-level feature space, not raw pixels). 3. A lightweight prediction head (one or two linear layers) maps \(\Delta F\) to a normalized 2D motion vector \((v_x, v_y)\) encoding direction and relative magnitude. 4. This auxiliary objective is jointly optimized with standard instruction tuning.
Reported results
| Benchmark | Baseline | With DeltaDirect | |---|---|---| | MoDirect-SynBench (synthetic) | 25.9% (chance) | 85.4% | | MoDirect-RealBench (real world) | near chance | +21.9 points, with no real-world fine-tuning data |
Crucially, standard video understanding performance is fully preserved — no "alignment tax".
The MoDirect dataset family
Broader implications
The post frames the result philosophically (echoing the Chinese Room argument): models manipulate symbols well but the binding between perception and language breaks at perceptual primitives like direction. DeltaDirect narrows this gap by making direction extraction an *intrinsic* representation-learning goal at the translation point between vision and language — like purifying water at the river's source rather than downstream.
Direction sense underpins navigation, prediction, and causal inference (which object will hit which wall, where the ball is heading). A model without it is like a navigation system with all map data but no sense of north. The takeaway: sometimes the deepest progress is not more parameters or data, but checking whether we've actually taught machines the most basic things.