[论文] EgoActionBench: Benchmarking Dense Action Captioning for Egocentric Hu...

论文概要

研究领域: CV 作者: Yujiao Shen, Jiahe Song, Haitian Li, Jingkang Yang, Ziwei Liu 发布时间: 2026-11-02 arXiv: 2611.03646

中文摘要

从人类数据学习是实现具身智能的可扩展路径，但这要求模型能从第一人称观察中精确理解细粒度的人类动作。然而现有基准在动作原子性、时间精度与手-动作关联上仍有不足。我们提出 EgoActionBench：将连续的人类操作表示为时间定位的原子动作，并关联到左手、右手或双手。该基准包含 1,407 段视频与 41,132 条人工精修的动作标注。为评估这种密集预测，我们提出一套评估协议：检索时间合格的候选、按内容一致性选取预测，并从语义保真度、时间重叠、动作精确率与召回率多维度打分，联合刻画「发生了什么、何时发生、由哪只手执行」。我们在 EgoActionBench 上评测了 20 个基线模型与系统，并对 Qwen3-VL-4B 与 LLaVA-OneVision-2-8B 两个骨干模型进行了监督微调+强化学习的改造。EgoAction 使两个骨干的 EgoActionScore 分别提升 15.9 和 23.2 分，且改造后的模型均超过所有开源基线。实验揭示了当前模型在动作原子性、时间精度、手-动作关联上的显著局限，凸显出通用视频理解与具身智能所需动作理解之间的鸿沟。

原文摘要

Learning from human data offers a scalable path toward embodied intelligence, yet it requires models to accurately understand fine-grained human actions from egocentric observations. However, existing benchmarks remain limited in action atomicity, temporal precision, and hand-action association. We introduce EgoActionBench, a benchmark that represents continuous human manipulation as temporally localized atomic actions associated with the left hand, right hand, or both hands. EgoActionBench contains 1,407 episodes with 41,132 human-refined action annotations. To evaluate such dense predictions, we propose an evaluation protocol that retrieves temporally eligible candidates, selects a prediction by content agreement, and scores semantic fidelity and temporal overlap alongside action precisi...

#论文 #arXiv #CV #小凯