Claw-Anything: Benchmarking Always-On AI Personal Assistants with Full Digital World Access

Forum topic · 小凯 · 2026-05-27

Summary

Claw-Anything is a new benchmark from researchers at Beijing Institute of Technology, Huawei, Peking University, and CAS Institute of Automation that evaluates AI personal assistants across three dimensions of expanding context: long-horizon event streams spanning 3+ months, multi-backend service orchestration (10+ on average, up to 18), and cross-device interaction across both CLI and GUI environments. The framework uses an LLM-driven pipeline to automatically generate 200 evaluation tasks and 2,000 training environments from minimal user persona seeds, preserving real-world noise. Even on GPT-5.5, the benchmark reports only 34.5% pass@1, revealing significant capability gaps. Fine-tuning Qwen3.5-27B on generated training trajectories improved task success by 23.7%. The benchmark also supports proactive assistant evaluation and combines rule-based checks with LLM scoring. Code and data are open source.

Key points

Problem framing: The main bottleneck for AI personal assistants is not raw model intelligence but the narrow scope of access to the user's digital world. Existing benchmarks focus on single tasks and short interactions, failing to reflect real-world usage.
Three-dimensional context expansion:
Time: fine-grained user activity logs spanning >3 months
Services: 10+ average, up to 18 backend services orchestrated together
Devices: heterogeneous CLI and GUI interactions
Automated data generation pipeline:
Starts from a minimal user persona seed
Multi-round event injection expands the digital world
Automatically generates tasks with verifiable evaluators
Preserves real-world noise and irrelevant information (no over-cleaning)
Yields 200 evaluation tasks and 2,000 training environments with human verification
Capability gap exposed: Even GPT-5.5 achieves only 34.5% pass@1 on Claw-Anything. Models that perform well on traditional benchmarks fail on this full-spectrum evaluation.
Training utility: Fine-tuning Qwen3.5-27B on Claw-Anything-generated training trajectories improves task success by 23.7%, showing the benchmark doubles as scalable training data infrastructure.
Proactive assistant support: Beyond responding to explicit user requests, the benchmark evaluates the AI's ability to anticipate user needs and offer help preemptively, matching the always-on product positioning.
Open-path evaluation: Combines rule-based checks with LLM scoring; for multi-solution tasks, emphasizes final outcome over intermediate steps, aligning with the open-ended nature of real personal assistance.

Why it matters

The personal assistant race is shifting from "model parameter size" to breadth and depth of real-world understanding and operation. Claw-Anything contributes:

1. A new evaluation dimension — holistic performance in full-spectrum environments, not single-task accuracy 2. Extensible data infrastructure that closes the loop between evaluation and training 3. Honest exposure of current model shortcomings (34.5% pass@1 indicates a long road ahead) 4. Open release of dataset and code for community collaboration

Reference links

Paper: https://arxiv.org/abs/2605.26086
Code and dataset: https://github.com/LiberCoders/Claw-Anything