A Decade of Vision-Language Model Accuracy and Visual-Cognitive Errors on Complex Social Behavior Images

Forum topic · 小凯 · 2026-07-14

Summary

A paper by Shravan Murlidaran and Miguel P. Eckstein (arXiv:2607.09654, July 10, 2026) tracks the evolution of vision-language models (VLMs) from 2017 to 2025 in scene description tasks. While prior evaluations largely relied on simple scenes like MS-COCO, the authors introduce the Complex Social Behavior (CSB) dataset, consisting of 100 images depicting intricate social interactions and behaviors. Nine models (four pre-MLLM and five MLLM) were analyzed. On CSB, accuracy improvements were far more pronounced than on MS-COCO. Pre-MLLMs scored well below human performance, whereas multimodal LLMs (MLLMs) reached parity with top human annotators. MLLMs nearly eliminated all error types, with one exception: occasional spatial dependency errors, where models rely on different image regions than humans do when interpreting complex social scenes. The findings suggest that while modern multimodal models largely match human-level accuracy on complex social perception, subtle visual-cognitive gaps remain in where models direct attention.

Paper Overview

Field: CV/AI Authors: Shravan Murlidaran, Miguel P. Eckstein arXiv: 2607.09654

Key Points

Vision-language models (VLMs) have made significant progress in visual reasoning over the past decade, but most evaluations use simple scenes such as MS-COCO.
The paper introduces the Complex Social Behavior (CSB) dataset, containing 100 images depicting complex social interactions and behaviors.
The study analyzes scene description progress from 2017 to 2025 across nine models: 4 pre-MLLM and 5 MLLM systems.
The CSB dataset shows more pronounced accuracy gains than MS-COCO, revealing larger gaps in earlier models on complex scenes.
Pre-MLLM accuracy was far below human performance, while MLLMs reached a level comparable to top human annotators.
MLLMs nearly eliminated all error types except spatial dependency errors — occasionally relying on different image regions than humans when interpreting scenes.

Conclusion

Modern multimodal LLMs largely match top human performance in describing complex social scenes, but subtle visual-cognitive differences remain in which image regions models depend on.

---

Paper link: https://arxiv.org/abs/2607.09654