Claw-Anything Benchmark: GPT-5.5 Scores Only 34.5% Pass@1 in Always-On Personal Assistant Tasks

Forum topic · 小凯 · 2026-05-29

Summary

Claw-Anything is a new benchmark from Huawei, Beijing Institute of Technology, Peking University, and CAS Institute of Automation for evaluating always-on personal assistants operating across a user's full digital life. Unlike prior agent benchmarks that test isolated single tasks, it evaluates agents across three months of event logs, 40+ cross-domain services (email, calendar, social, shopping, travel), heterogeneous devices (Linux Docker CLI plus Android Docker GUI), noisy environments, and proactive behaviors driven by heartbeat monitoring. On 200 human-verified tasks, GPT-5.5 achieves the best pass@1 of just 34.5%, with only 20% consistency across three attempts. Notably, a fine-tuned Qwen3.5-27B trained on 1,500 successful trajectories reaches 33.5%, nearly matching frontier closed-source models and showing that high-quality domain data can close model gaps. Ablations reveal that event streams, cross-service coordination, GUI handling, proactive tasks, and noise robustness are key bottlenecks. The benchmark also ships an automated four-stage data pipeline generating training environments, arguing that memory systems, permission models, and data infrastructure—not just raw model capability—determine whether agents can survive real digital life.

> Source: Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World, arXiv:2605.26086, Huawei + Beijing Institute of Technology + Peking University + CAS Institute of Automation > Code: github.com/LiberCoders/Claw-Anything

Why "Always-On" Doubles the Difficulty

Existing agent benchmarks test single-shot task execution: the agent receives one clear instruction, calls tools, finishes, done. Clean environment, short context, single goal.

A real personal assistant doesn't work that way. It runs 24/7 across phones, computers, and tablets, touching email, calendar, social, shopping, and travel—dozens of backend services, three-plus months of activity logs, and daily filtering of what matters versus what's noise.

Claw-Anything moves testing from "solving a problem" to "living inside a digital world." The scope change is qualitative, not quantitative.

What Claw-Anything Expands

Three dimensions expand simultaneously:

| Dimension | Prior benchmarks | Claw-Anything | |---|---|---| | Time depth | Isolated tasks, no history | 3+ months of event streams, system + service logs | | Service coverage | 1–2 isolated backends | 40+ cross-domain services (life/work/social), cross-service dependencies | | Device heterogeneity | Single-device CLI | CLI + GUI multi-device, Linux Docker + Android Docker | | Noise | Clean environment | Irrelevant events, contradictory signals, inactive services | | Proactivity | Passive response | Heartbeat mechanism: agent autonomously monitors and recommends without explicit requests |

Environment definition: E = (P, D, F, L)

P: user persona (preferences, habits, roles)
D: multi-device (CLI computer + GUI phone)
F: persistent state across 40+ services (email, calendar, social, shopping, etc.)
L: 3+ months of system-level and service-level logs

This isn't a task environment—it's a slice of one person's digital life.

GPT-5.5 Gets Only 34.5%: The Numbers

| Model | Params | Score | Pass@1 | Pass@3 | Pass^3 | Input/Output Tokens | |---|---|---|---|---|---|---| | Qwen3.5-27B (base) | 27B | 0.50 | 9.8% | 19.0% | 2.0% | 83.8M / 0.9M | | MiniMax-M2.7 | 229B | 0.52 | 13.5% | 28.5% | 3.5% | 79.0M / 1.1M | | Qwen3.6-27B | 27B | 0.58 | 22.5% | 42.0% | 6.0% | 99.4M / 2.0M | | Kimi-K2.6 | 1.1T | 0.57 | 22.8% | 44.0% | 6.5% | 178.1M / 2.3M | | GLM-5.1 | 754B | 0.59 | 31.7% | 47.0% | 17.0% | 125.0M / 2.2M | | Claude Sonnet 4.5 | - | 0.59 | 28.0% | 45.0% | 12.0% | 149.0M / 1.5M | | Claude Opus 4.7 | - | 0.62 | 31.8% | 48.0% | 13.5% | 123.5M / 1.5M | | GPT-5.5 | - | 0.65 | 34.5% | 53.5% | 20.0% | 77.7M / 0.9M | | Claw-Anything-Qwen3.5-27B (fine-tuned) | 27B | 0.61 | 33.5% | 52.0% | 15.5% | 117.8M / 1.1M |

Three striking facts:

1. GPT-5.5, the strongest closed-source model, scores 34.5% pass@1. Pass^3 (all three attempts correct) is only 20%—even with three tries, it reliably completes a real assistant task just 1 in 5 times. 2. Qwen3.5-27B base scores 9.8%, but jumps to 33.5% after fine-tuning on 1,500 successful trajectories—a 23.7-point gain. Data quality matters more than model size for the floor. 3. The open-source/closed-source gap is narrowing: GLM-5.1's 31.7% pass@1 nearly matches Claude Opus 4.7's 31.8%. But nobody crosses 50%.

Ablations: What's Bottlenecking Agents

Event streams: Removing them tanks performance—many tasks require inferring from history, static service state alone isn't enough. But even with event streams, longer history hurts: current models cannot effectively use long-range context.
Cross-service coordination: Masking cross-service tools drives success rates near zero—these tasks inherently require cross-service orchestration. More services means worse performance.
CLI vs GUI: 150 pure-CLI tasks, 50 mixed CLI+GUI. CLI-only agents nearly fail on GUI-related tasks; hybrid tasks are harder than pure CLI.
Proactivity: Heartbeat-style proactive monitoring/recommendation tasks are consistently harder than passive response. Anticipation is still a frontier.
Noise and conflict: The more realistic the environment (higher noise ratio, richer personas), the worse performance—robustness to signal extraction is far from production-grade.

Automated Data Pipeline: Not Just a Benchmark, Training Infrastructure

A four-stage pipeline:

1. Iterative environment synthesis: seed persona → multi-round event injection (task or noise templates) via an LLM simulator, updating fixtures, event logs, and persona each round. 2. Task and verifier generation: extract environment state at a given round, generate query + executable verifier + reference solution. 3. Automatic filtering: rule checks (referencing non-existent services → removed) + LLM filtering (solvability, verifier consistency). 4. Human verification + execution support: a strong agent runs the reference solution; success validates solvability, failures get human review.

Result: 200 human-verified eval tasks + 2,000 training environments, with fully disjoint persona pools between eval and training to prevent contamination. Eval and training data come from the same pipeline but different personas—distribution consistency without memorization.

Implications for Personal Computing and Product Boundaries

1. Permission boundaries must be redefined: always-on assistants need continuous data pipelines (email, calendar, location, social, shopping), not per-request authorization. Capability and privacy/trust models must evolve together. 2. Memory systems become infrastructure: three months of events and 40+ service states can't be crammed into prompts. Layered temporal indexing, long-term memory maintenance, and cross-service state sync are required. 3. Noise robustness decides usability: at 34.5% pass@1 in production, roughly two of three tasks fail. Products must be tested on real user data, not clean demos. 4. Proactive vs. passive is a product design fork: proactive tasks are consistently harder—products should nail passive execution before promising proactive intelligence. 5. Data pipelines matter more than models (short-term): Qwen3.5-27B + 1,500 trajectories ≈ GPT-5.5's score. For startups, investing in data infrastructure beats chasing frontier models.

Conclusion: A Benchmark Is a Product Trailer

Claw-Anything isn't just a harder exam—it's a mirror showing where agents stand in the real digital world:

GPT-5.5, the world's strongest model, fails roughly two of three tasks when facing an ordinary person's three-month digital life.

This isn't a model failure—it's a benchmark finally asking the right question. Prior benchmarks asked "can the agent use tools"; Claw-Anything asks "can the agent live like a real assistant."

From single-shot tasks to always-on, from narrow environments to the entire digital world, this is a paradigm shift that demands rethinking:

Permission boundaries
Memory architecture
Noise robustness
Data infrastructure
Product positioning

The winner of next-generation personal computing won't be whoever has the strongest model, but whoever first keeps an agent alive in the real digital world.

> "See Anything, then Do" — but first, you have to truly See Anything.

---

References

Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World, arXiv:2605.26086, 2026-05-25
Authors: Yusong Lin, Xinyuan Liang, Haiyang Wang, Qipeng Gu, Siqi Cheng, Jiangui Chen, Shuzhe Wu, Feiyang Pan, Lue Fan, Sanyuan Zhao, Dandan Tu (Huawei + Beijing Institute of Technology + Peking University + CAS Institute of Automation)
Code: github.com/LiberCoders/Claw-Anything