WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation in Native Runtimes

Forum topic · 小凯 · 2026-05-13

Summary

WildClawBench (arXiv:2505.07235) is a benchmark for evaluating LLM and vision-language model agents that act on a user's behalf through command-line interface (CLI) harnesses. Unlike most agent benchmarks that rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, WildClawBench is a native-runtime benchmark consisting of 60 human-authored, bilingual, multimodal tasks spanning realistic long-horizon workflows. It addresses the open question of whether agents can complete real-world, extended work in the runtimes where they are actually deployed. Authored by Shuangrui Ding, Xuanlang Dai, and Long Xing, the paper was released on May 9, 2025, in the NLP domain. The benchmark provides a more faithful evaluation setting by testing agents directly in their native runtime environments with diverse, human-written tasks rather than artificial sandbox conditions.

Paper Overview

Research Area: NLP Authors: Shuangrui Ding, Xuanlang Dai, Long Xing Published: 2025-05-09 arXiv: 2505.07235

Abstract

Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed.

This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning realistic long-horizon agent workloads.

Key Contributions

Native-runtime evaluation: Tasks run in the actual runtime environments where agents are deployed, rather than synthetic sandboxes
Long-horizon tasks: 60 human-authored tasks designed to test extended, realistic workflows
Bilingual and multimodal: Task suite covers multiple languages and modalities
Beyond final-answer checks: Moves past simple answer checking toward evaluating realistic agent behavior