GPT-5.5 Scores Just 34.5% on Claw-Anything: A New Benchmark for Always-On Personal Assistants
> Source: *Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World*, arXiv:2605.26086 — Huawei + Beijing Institute of Technology + Peking University + CAS Institute of Automation > Code: github.com/LiberCoders/Claw-Anything
Why "Always-On" Doubles the Difficulty
Existing agent benchmarks test single-shot task execution: an agent gets a clear instruction, calls tools, finishes, done. Clean environment, short context, single goal.
But real personal assistants don't work that way. They are online 24/7, spanning phones, computers, and tablets, handling email, calendars, social media, shopping, and travel — dozens of backend services, three-plus months of activity logs, and daily decisions about which signals matter and which are noise.
Claw-Anything moves testing from "solving a problem" to "living inside a digital world." The scope change is qualitative, not quantitative.
How Broad Is the Scope?
The benchmark expands three dimensions simultaneously:
| Dimension | Prior benchmarks | Claw-Anything | |------|-------------|---------------| | Temporal depth | Isolated tasks, no history | 3+ months of event streams, system logs + service history | | Service coverage | 1–2 isolated backends | 40+ cross-domain services, with cross-service dependencies | | Device heterogeneity | Single-device CLI | CLI + GUI multi-device, Linux Docker + Android Docker | | Noise | Clean environment | Irrelevant events, conflicting signals, inactive services | | Proactivity | Reactive only | Heartbeat mechanism — agent monitors and recommends without explicit requests |
The environment is defined as E = (P, D, F, L):
- P: user persona (preferences, habits, roles)
- D: multiple devices (CLI desktop + GUI phone)
- F: persistent state across 40+ services (email, calendar, social, shopping, etc.)
- L: 3+ months of system-level and service-level logs
- Event streams: Removing them collapses performance — many tasks require inferring from history; static service state alone isn't enough. But even with event streams, longer history hurts — current models don't exploit long-range context effectively.
- Cross-service coordination: Blocking cross-service tools drops success to near zero — these tasks fundamentally require cross-service orchestration. More services means worse performance.
- CLI vs GUI: 150 pure-CLI tasks, 50 mixed CLI+GUI. Restricting agents to CLI makes GUI tasks nearly impossible. Mixed tasks are harder; heterogeneous interface coordination is the hard part.
- Proactivity: Heartbeat-style tasks (agent monitors and recommends unprompted) are consistently harder than reactive responses. Anticipation remains a frontier.
- Noise and conflict: The more realistic the environment (higher noise ratio, richer personas), the lower the performance — current agents' robustness to signal extraction is far from production grade.
- *Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World*, arXiv:2605.26086
- Code: github.com/LiberCoders/Claw-Anything
This is not a task environment — it is a slice of one person's digital life.
The Numbers: GPT-5.5 Gets 34.5%
| Model | Params | Score | Pass@1 | Pass@3 | Pass^3 | Tokens (in/out) | |------|--------|-------|--------|--------|--------|----------------| | Qwen3.5-27B (base) | 27B | 0.50 | 9.8% | 19.0% | 2.0% | 83.8M / 0.9M | | MiniMax-M2.7 | 229B | 0.52 | 13.5% | 28.5% | 3.5% | 79.0M / 1.1M | | Qwen3.6-27B | 27B | 0.58 | 22.5% | 42.0% | 6.0% | 99.4M / 2.0M | | Kimi-K2.6 | 1.1T | 0.57 | 22.8% | 44.0% | 6.5% | 178.1M / 2.3M | | GLM-5.1 | 754B | 0.59 | 31.7% | 47.0% | 17.0% | 125.0M / 2.2M | | Claude Sonnet 4.5 | - | 0.59 | 28.0% | 45.0% | 12.0% | 149.0M / 1.5M | | Claude Opus 4.7 | - | 0.62 | 31.8% | 48.0% | 13.5% | 123.5M / 1.5M | | GPT-5.5 | - | 0.65 | 34.5% | 53.5% | 20.0% | 77.7M / 0.9M | | Claw-Anything-Qwen3.5-27B (fine-tuned) | 27B | 0.61 | 33.5% | 52.0% | 15.5% | 117.8M / 1.1M |
Three striking facts:
1. GPT-5.5, the strongest closed-source model, gets 34.5% pass@1. Pass^3 (all three attempts correct) is only 20% — even with three chances, it reliably completes a realistic assistant task just 1 in 5 times.
2. Qwen3.5-27B base starts at 9.8%, but jumps to 33.5% after fine-tuning on 1,500 successful trajectories — a 23.7-point gain. Data quality matters more than model size for the floor.
3. The open/closed-source gap is narrowing: GLM-5.1's 31.7% pass@1 nearly matches Claude Opus 4.7's 31.8%. But nobody passes 50%.
Ablations: What's the Bottleneck?
Automated Data Pipeline: Benchmark Plus Training Infrastructure
Hand-building 200 realistic digital-world tasks doesn't scale. Claw-Anything's pipeline has four stages:
1. Iterative environment synthesis: Starting from a persona seed, multi-round event injection (task or noise templates) grows the digital world via an LLM simulator, updating fixtures, event logs, and persona each round. 2. Task and verifier generation: Extract environment state at a specific round; generate query + executable verifier + reference solution. 3. Automatic filtering: Rule checks (remove tasks referencing nonexistent services) + LLM filtering (solvability, verifier consistency). 4. Human verification + execution support: A strong agent executes the reference solution; success validates solvability, failure triggers manual review.
Result: 200 human-verified eval tasks + 2,000 training environments, with personas fully isolated between eval and training to prevent contamination. Eval and training data come from the same pipeline but different persona pools — consistent distribution without memorization.
Implications for Personal Computing and Product Boundaries
1. Permission boundaries must be redefined. Always-on assistants need continuous access to email, calendar, location, social, and shopping data — persistent data pipelines, not one-off authorizations. Stronger capability demands deeper digital access; privacy and trust models must evolve in step.
2. Memory becomes infrastructure, not a nice-to-have. Three months of events, 40+ service states, cross-device traces — prompt engineering can't hold this. Without hierarchical temporal indexing, long-term memory maintenance, and cross-service state sync, an always-on assistant is blind.
3. Noise robustness determines product viability. The more realistic the environment, the worse agents perform. Products must be tested on real user data, not clean demos. A 34.5% pass@1 means "two failures out of three tasks" — unacceptable in production.
4. Proactive vs reactive is a product-design fork. The benchmark evaluates them separately, and proactive is always harder. Next-gen products should nail reactive execution before pursuing proactive intelligence.
5. The data pipeline matters more than the model (for now). Qwen3.5-27B + 1,500 trajectories = 33.5%, nearly matching GPT-5.5's 34.5%. For startups, investing in data pipelines beats chasing frontier models.
Conclusion: A Benchmark Is a Product Trailer
Claw-Anything is more than a harder exam — it's a mirror showing where agents actually stand in the real digital world:
GPT-5.5, the world's strongest model, facing an ordinary person's three months of digital life, fails two out of three tasks.
This isn't a weak model — it's a benchmark finally asking the right question. Previous benchmarks tested whether agents can use tools; Claw-Anything tests whether an agent can live like a real human assistant.
The shift from single-shot to always-on, from narrow environments to the entire digital world, is paradigm-level. It demands rethinking: permission boundaries, memory architecture, noise robustness, data infrastructure, and product positioning.
The winner of next-generation personal computing won't be the model with the strongest scores — it will be whoever first keeps an agent alive in the real digital world.
References