PerceptionComp: A Manually Annotated Video Benchmark for Complex Perception-Centric Reasoning

Forum topic · 小凯 · 2026-03-31

Summary

PerceptionComp is a fully hand-annotated benchmark introduced for complex, long-horizon, perception-centric video reasoning. Its questions are designed so that no single video moment suffices: each answer requires combining multiple temporally separated pieces of visual evidence under conjunctive and sequential logic, spanning perceptual subtasks such as objects, attributes, relations, locations, actions, and events, and demanding semantic recognition, visual correspondence, temporal reasoning, and spatial reasoning. The benchmark comprises 1,114 highly complex questions over 279 videos from diverse domains, including city walking tours, indoor house tours, video games, and extreme outdoor sports. Human studies show participants spend considerably more time than on prior benchmarks, and accuracy drops to near chance (18.97%) when re-watching is disallowed. State-of-the-art multimodal large language models also struggle: the best model evaluated, Gemini-3-Flash, achieves only 45.96% accuracy in a five-choice setting, while open-source models remain below 40%. These results indicate that perception-centric long-horizon video reasoning remains a major bottleneck. Paper: arXiv 2503.23716.

Overview

Field: NLP Authors: Shaoxuan Li, Zhixuan Zhao, Hanze Deng Published: 2025-03-30 arXiv: 2503.23716

Introduction

We introduce PerceptionComp, a manually annotated benchmark for complex, long-horizon, perception-centric video reasoning.

PerceptionComp is designed so that no single moment is sufficient: answering each question requires multiple temporally separated pieces of visual evidence and compositional constraints under conjunctive and sequential logic, spanning perceptual subtasks such as objects, attributes, relations, locations, actions, and events, and requiring skills including semantic recognition, visual correspondence, temporal reasoning, and spatial reasoning.

Benchmark Construction

The benchmark contains 1,114 highly complex questions over 279 videos from diverse domains, including:

City walking tours
Indoor house tours
Video games
Extreme outdoor sports

All questions are 100% manually annotated.

Human Study

PerceptionComp demands substantial test-time thinking and repeated perceptual steps:

Participants spend considerably longer than on previous benchmarks.
When re-watching is disallowed, accuracy drops to near chance (18.97%).

Model Evaluation

State-of-the-art multimodal large language models perform substantially worse on PerceptionComp than on existing benchmarks:

The best model in the evaluation, Gemini-3-Flash, reaches only 45.96% accuracy in a five-choice setting.
Open-source models remain below 40%.

Conclusion

These results show that perception-centric long-horizon video reasoning remains a major bottleneck. The authors hope PerceptionComp will help drive progress in perceptual reasoning.

Paper link: arXiv:2503.23716