论文 Overview
- Field: NLP
- Authors: Zhekai Chen, Chengqi Duan, Kaiyue Sun
- Published: 2026-07-10
- arXiv: 2507.08180
- 400 bilingual real-world tasks built on the five capabilities
- Evaluation in live Docker containers with fine-grained, step-by-step completion checkpoints, unlike prior benchmarks that depend on static pre-recorded answers
- A closed-loop evaluation strategy consisting of an execution agent, a hidden supervising agent, and a user agent, which simulates realistic multi-turn human feedback without leaking grading criteria
Background
The rapid development of large language models (LLMs) and multimodal LLMs has accelerated the emergence of proactive agents that can operate everyday tools and assist users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively because they often rely on sandboxed environments and single-turn evaluation paradigms. Their scenario-based task taxonomies also mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures.
Key Contributions of UniClawBench
UniClawBench is the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings.
Five foundational capabilities assessed:
1. Skill Usage 2. Exploration 3. Long-context Reasoning 4. Multimodal Understanding 5. Cross-platform Coordination
Benchmark design:
Significance
By decoupling tasks by capability rather than scenario, UniClawBench enables finer-grained diagnosis of why proactive agents fail, offering a more rigorous and realistic evaluation methodology for the next generation of tool-using, multimodal agents.
---
*Auto-collected on 2026-07-11*