[论文] UniClawBench: A Universal Benchmark for Proactive Agents on Real-World...

论文概要

研究领域: NLP
作者: Zhekai Chen, Chengqi Duan, Kaiyue Sun
发布时间: 2026-07-10
arXiv: 2507.08180

中文摘要

大语言模型和多模态大语言模型的快速发展加速了主动代理（proactive agents）的出现，它们能够操作日常工具并在真实环境中协助用户。然而，现有基准测试难以有效评估此类代理，因为它们往往依赖沙箱环境和单轮评估范式。此外，其基于场景的任务分类法在同一任务类别中混合了多种模型能力，难以识别代理失败的根本原因。为应对这些局限，我们引入UniClawBench，首个以能力为导向的基准测试，专为动态真实环境中评估主动代理而设计。UniClawBench围绕五项基础模型能力构建：技能使用、探索、长上下文推理、多模态理解和跨平台协调。基于这些能力，我们设计了400个双语真实世界任务。与以往依赖静态预录答案的基准不同，我们的基准在实时Docker容器中使用细粒度的逐步完成检查点来评估代理。此外，我们设计了包含执行代理、隐藏监督代理和用户代理的闭环评估策略，模拟真实的多轮人类反馈而不泄露评分标准。

原文摘要

The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, E...

自动采集于 2026-07-11

#论文 #arXiv #NLP #小凯