Diagnosing Directional Motion Blindness in Video-LLMs: DeltaDirect Boosts Motion Direction Accuracy to 85.4%

Forum topic · 小凯 · 2026-05-23

Summary

A new paper by Jongseo Lee, Hyuntak Lee, and Sunghun Kim (arXiv:2505.17389, May 2025) identifies "directional motion blindness" in Video-LLMs: despite advances in temporal video understanding, most models perform near chance when asked which way a single object moved on screen (left, right, up, or down). The authors trace the pipeline and find that motion direction is linearly decodable in the visual encoder, projector, and LLM hidden states, but the readout fails to bind this signal to the correct linguistic answer—a "direction binding gap." Synthetic instruction tuning narrows the gap in-domain, yet visual complexity weakens the signal, limiting generalization. To address this, the team introduces the MoDirect dataset family for motion-direction instruction tuning and evaluation, plus DeltaDirect, a diagnostic-driven projector-level objective that predicts normalized 2D motion vectors from feature differences of adjacent frames. On MoDirect-SynBench, DeltaDirect tuning lifts motion direction accuracy from 25.9% to 85.4%; on MoDirect-RealBench, it improves real-world accuracy by 21.9 percentage points without real-world fine-tuning data, while preserving standard video understanding performance.

Paper Overview

Field: Computer Vision Authors: Jongseo Lee, Hyuntak Lee, Sunghun Kim Published: 2025-05-23 arXiv: 2505.17389

Abstract (English translation)

Video large language models (Video-LLMs) have made rapid progress in temporal video understanding, yet many fail at a basic perceptual primitive: signed, image-plane motion direction. In simple videos where a single object moves left, right, up, or down, most Video-LLMs perform near chance, and the few cases exceeding chance can largely be attributed to prediction bias rather than genuine direction understanding. The authors call this failure directional motion blindness.

By tracing motion direction information through the Video-LLM pipeline, they find that direction is linearly decodable in the visual encoder, projector, and LLM hidden states—but the readout mechanism fails to bind this signal to the correct linguistic answer option, exposing a direction binding gap. While synthetic motion-direction instruction fine-tuning reduces this gap on the source domain, motion-direction concept vector analysis shows that visual complexity attenuates the signal magnitude, limiting cross-domain generalization.

Contributions

MoDirect dataset family: datasets for motion-direction instruction fine-tuning and evaluation, including MoDirect-SynBench and MoDirect-RealBench.
DeltaDirect: a diagnostic-driven, projector-level objective that predicts normalized 2D motion vectors from feature differences between adjacent frames.

Results

On MoDirect-SynBench, instruction fine-tuning with DeltaDirect raises motion direction accuracy from 25.9% to 85.4%.
On MoDirect-RealBench, DeltaDirect improves real-world motion direction accuracy by 21.9 percentage points without any real-world fine-tuning data, while preserving standard video understanding performance.

---

*Auto-collected on 2026-05-23.*