UniClawBench: A Universal Benchmark for Proactive Agents in Real-World Environments

Forum topic · 小凯 · 2026-07-12

Summary

UniClawBench is the first capability-driven benchmark for evaluating proactive agents built on large language models and multimodal LLMs in dynamic, real-world environments. Unlike existing benchmarks that rely on sandboxed environments and single-turn evaluation, UniClawBench organizes tasks around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. The benchmark includes 400 bilingual real-world tasks evaluated inside live Docker containers with fine-grained step-by-step completion checkpoints rather than static preset answers. A closed-loop evaluation strategy—an execution agent, a hidden supervisor agent, and a user agent—simulates realistic multi-turn human feedback without leaking grading criteria. The authors evaluate state-of-the-art models across multiple agent frameworks to disentangle foundational model capabilities from framework-level design choices, showing how both jointly shape real-world performance. Developed by researchers including Zhekai Chen and Xihui Liu, the benchmark and code are publicly available at https://github.com/HKU-MMLab/UniClawBench. The paper appeared on arXiv as 2607.08768.

Paper Overview

Field: NLP
Authors: Zhekai Chen, Chengqi Duan, Kaiyue Sun, Bohao Li, Yuqing Wang, Manyuan Zhang, Xihui Liu
Published: 2026-07-09
arXiv: 2607.08768

Abstract

The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures.

To address these limitations, the authors introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings.

Key Features

Five foundational capabilities: UniClawBench is built around Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination.
400 bilingual real-world tasks designed around these capabilities.
Live evaluation: Unlike benchmarks relying on static preset answers, agents are assessed in real-time Docker containers using fine-grained, step-by-step completion checkpoints.
Closed-loop evaluation strategy: An execution agent, a hidden supervisor agent, and a user agent simulate realistic multi-turn human feedback without leaking the grading rubric.
Capability vs. framework analysis: State-of-the-art models are evaluated under multiple agent frameworks to separate foundational model capabilities from framework-level design choices.

Through comprehensive comparisons across models and frameworks, the paper demonstrates how foundational model capabilities and agent framework design jointly shape performance in real-world environments.

Resources

Code and benchmark: https://github.com/HKU-MMLab/UniClawBench
Paper: https://arxiv.org/abs/2607.08768