Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs

Forum topic · 小凯 · 2026-05-24

Summary

A forum post discusses a paper by Jongseo Lee, Hyuntak Lee, and Sunghun Kim (KHU-VLL, Kyung Hee University) revealing that most video large language models (Video-LLMs) perform at near-chance (~25%) accuracy when asked whether an object moved left, right, up, or down—a phenomenon termed Directional Motion Blindness. Layer-wise linear probing shows motion direction information remains linearly decodable through the vision encoder, projector, and LLM hidden states, but fails at binding perception to language labels: the 'Direction Binding Gap.' Synthetic motion-direction instruction tuning improves accuracy only on simple synthetic scenes, failing to generalize to real-world footage because visual complexity weakens the direction signal, as shown by motion direction concept vector analysis. The authors propose DeltaDirect, an auxiliary objective at the projector level that predicts a normalized 2D motion vector from differences between adjacent frame features, jointly optimized with instruction tuning. Results: accuracy rises from 25.9% to 85.4% on MoDirect-SynBench and improves by 21.9 points on real-world benchmarks without any real-world fine-tuning data, while preserving standard video understanding performance. Code is released at https://github.com/KHU-VLL/DeltaDirect.

Overview

This post reviews the paper *"Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs"* by Jongseo Lee, Hyuntak Lee, and Sunghun Kim (KHU-VLL, Kyung Hee University, arXiv cs.CV).

The Problem: Directional Motion Blindness

Most state-of-the-art Video-LLMs can summarize films and answer complex video-QA questions, yet they perform at near-chance accuracy (~25%) when judging whether an object moved left, right, up, or down—even in trivial scenes with a plain background and a single object. The paper names this systematic failure Directional Motion Blindness.

Diagnosis: The Direction Binding Gap

The authors trace motion direction information through the Video-LLM pipeline using linear probes:

Vision encoder: motion direction is fully preserved and linearly decodable.
Projector: the signal survives compression and modality alignment.
LLM hidden states: direction remains linearly accessible even after all Transformer layers.
Language output: the model fails completely (~25%, chance level).

The information exists in the model's internals, but the model cannot bind the perceptual concept of "direction" to the correct language labels. The authors call this the Direction Binding Gap—a binding failure, not an information loss.

Why Naive Fine-Tuning Fails

Synthetic motion-direction instruction tuning (colored blocks moving on plain backgrounds with directional QA) improves accuracy only on the synthetic source domain. On real-world videos, accuracy collapses back to baseline. Motion Direction Concept Vector Analysis shows why: as visual complexity increases, the direction signal's alignment with language labels is diluted and weakened, limiting cross-domain generalization.

The Fix: DeltaDirect

DeltaDirect is a diagnostic-driven auxiliary objective applied at the projector level:

1. Extract features $$F_t$$ and $F_{t+1}$ from adjacent frames via the vision encoder. 2. Compute the feature difference $\Delta F = F_{t+1} - F_t$ in high-level feature space. 3. A lightweight prediction head maps $\Delta F$ to a normalized 2D motion vector $$(v_x, v_y)$$ encoding direction and relative magnitude. 4. The objective is jointly optimized with standard instruction tuning.

Results

MoDirect-SynBench: baseline 25.9% → 85.4% with DeltaDirect.
MoDirect-RealBench: +21.9 percentage points on real-world motion direction accuracy, with no real-world fine-tuning data required.
Standard video understanding performance is fully preserved (no "alignment tax").

Significance

The paper suggests Video-LLMs are never explicitly trained on atomic, directional motion annotations—so direction binding never develops. DeltaDirect embeds motion-difference sensitivity directly into the visual-language translation layer, making direction a geometrically meaningful, controllable dimension of the representation space. Direction sense underpins navigation, prediction, and causal reasoning, making this a foundational step from "seeing" motion to understanding it.

Resources:

Code: https://github.com/KHU-VLL/DeltaDirect
Datasets: MoDirect-SynBench (synthetic training/eval) and MoDirect-RealBench (real-world generalization eval)