EgoActionBench: Benchmarking Dense Action Captioning for Egocentric Videos

Forum topic · 小凯 · 2026-11-03

Summary

EgoActionBench (arXiv:2611.03646) is a new benchmark for dense action captioning in egocentric video, aimed at embodied intelligence research. It represents continuous human manipulation as temporally localized atomic actions linked to the left hand, right hand, or both hands, comprising 1,407 episodes with 41,132 human-refined action annotations. The authors propose an evaluation protocol that retrieves temporally eligible candidates, selects predictions by content agreement, and scores semantic fidelity, temporal overlap, and action precision/recall to jointly assess what happened, when, and with which hand. Evaluations cover 20 baseline models and systems, plus supervised fine-tuning and reinforcement learning applied to Qwen3-VL-4B and LLaVA-OneVision-2-8B backbones. The resulting EgoAction model improves EgoActionScore by 15.9 and 23.2 points respectively, outperforming all open-source baselines. Experiments reveal significant current limitations in action atomicity, temporal precision, and hand-action association, highlighting the gap between general video understanding and the action understanding needed for embodied AI.

论文概要 (Paper Overview)

Research area: Computer Vision Authors: Yujiao Shen, Jiahe Song, Haitian Li, Jingkang Yang, Ziwei Liu Published: 2026-11-02 arXiv: 2611.03646

Abstract

Learning from human data offers a scalable path toward embodied intelligence, yet it requires models to accurately understand fine-grained human actions from egocentric observations. However, existing benchmarks remain limited in action atomicity, temporal precision, and hand-action association. The authors introduce EgoActionBench, a benchmark that represents continuous human manipulation as temporally localized atomic actions associated with the left hand, right hand, or both hands. EgoActionBench contains 1,407 episodes with 41,132 human-refined action annotations.

To evaluate such dense predictions, the paper proposes an evaluation protocol that retrieves temporally eligible candidates, selects a prediction by content agreement, and scores semantic fidelity and temporal overlap alongside action precision and recall — jointly characterizing "what happened, when it happened, and which hand performed it."

Key Results

20 baseline models and systems evaluated on EgoActionBench.
Supervised fine-tuning + reinforcement learning applied to two backbones: Qwen3-VL-4B and LLaVA-OneVision-2-8B.
The resulting EgoAction models improve EgoActionScore by 15.9 and 23.2 points respectively, and both outperform all open-source baselines.
Experiments reveal significant limitations of current models in action atomicity, temporal precision, and hand-action association, underscoring the gap between general video understanding and the action understanding required for embodied intelligence.