TerminalWorld: 80,870 Real Terminal Recordings Show the Best AI Solves Fewer Than Two-Thirds of Real Command-Line Tasks
How many everyday command-line tasks can Claude Opus 4.7—one of the strongest AI models available—actually complete at a terminal?
The answer: less than two-thirds.
That is the core finding of TerminalWorld, a new paper from UCL (University College London) together with Nanjing University and Tencent. The team collected 80,870 real programmer terminal recordings from the internet, converted them into 1,530 executable evaluation tasks via an automated pipeline, and put 8 frontier LLMs and 6 terminal agents to the test.
The result? The strongest model, Claude Opus 4.7, achieved only a 62.5% pass rate. Models that score 80+ on expert-designed benchmarks stumble badly on real-world tasks.
A Counter-Intuitive Finding: High Exam Scores ≠ Real-World Skill
TerminalWorld's results correlate with the popular Terminal-Bench benchmark at only Pearson r = 0.20—statistically, almost no relationship.
GPT-5.5 scores 82.7% on Terminal-Bench, but only 53.5% on TerminalWorld's real tasks. Meanwhile, open-source Kimi K2.6, at just 66.7% on Terminal-Bench, scores 57.5% on TerminalWorld—beating both GPT-5.5 and Gemini 3.1 Pro.
Why? Expert-designed benchmarks and the real world are two different things. Experts naturally design "hard" problems with clever solutions requiring deep reasoning. But real terminal work is often: configuring Kubernetes clusters, writing CI/CD pipelines, debugging Docker networking, provisioning infrastructure with terraform. These tasks aren't necessarily "hard," but they require knowing the right tools and workflows—navigating an open action space.
Like a math olympiad gold medalist who is unbeatable at puzzle problems but may not know where the wrench is when fixing a pipe.
How TerminalWorld Was Built
The most ingenious part of the paper is its data construction method. Think of a cooking competition: the traditional approach asks chefs to invent their own challenges; TerminalWorld instead finds 80,870 real cooking videos and reverse-engineers each into an exam question.
The four-step pipeline:
Step 1: Collect recordings. asciinema is a platform where programmers share terminal recordings—like game replays for the command line. 80,870 recordings were collected.
Step 2: Distill tasks. Real recordings are full of typos, retries, and boring ls and cat commands. Using an LLM (Claude Sonnet 4.6), the pipeline distills two things from the noise: a task description (goal only, no steps) and a reference solution (a clean command script). Filtering left 9,492 high-quality recordings.
Step 3: Reproduce environments. Recordings capture commands but not what was installed on the programmer's machine. Claude Code acted as "environment engineer," inferring dependencies from the reference solution, then writing Dockerfiles, building images, running containers, and replaying commands. If it failed, fix it; if unfixable, discard. 5,035 tasks had environments successfully reproduced.
Step 4: Generate tests. A "triple adjudication" mechanism ensures automatic grading quality:
- All-pass adjudication: running the reference solution must pass all tests (prevents false negatives)
- No-op adjudication: doing nothing must fail all tests (prevents hollow tasks)
- Partial adjudication: running an incomplete solution must fail at least one test (prevents overly lenient tests)
The pipeline's key strength: it is fully automated and can run continuously. New recordings are uploaded to asciinema daily, so TerminalWorld can be refreshed regularly, always tracking current programmer practice. This is what the paper means by "authentic and scalable by construction."
The Efficiency Paradox: The Harder You Try, the More You Fail?
The most thought-provoking finding is the "Efficiency Paradox."
In TerminalWorld, task success rate is negatively correlated with resource consumption—more tokens and more turns mean a higher chance of failure. Failed tasks consumed 3.3x more tokens on average than successful ones, accounting for 63% of total cost but only 43% of attempts.
GPT-5.5 and MiniMax M2.7 are typical "hard workers"—they consume far more tokens and turns than other models yet achieve lower pass rates.
The cause: real terminal tasks present an open action space. Unlike coding problems with clear inputs and outputs, terminal tasks may require installing software, configuring environments, running commands, checking results—with countless choices at each step. Without reliable planning and stopping strategies, the AI keeps exploring, going further and spending more without getting closer to the answer.
Like someone in a maze with no idea where the exit is: walking more doesn't mean getting closer to the exit.
Do AI and Humans Take the Same Path?
Because every task comes from a real human recording, TerminalWorld enables direct comparison of AI and human solution paths.
The finding: AI and humans almost never take the same path. Median command-set overlap is only 21.4%.
For example, in a network packet analysis task (extracting HTTP Basic Auth credentials from a pcap file), a human used ettercap to replay and parse the capture, while the AI used tshark plus a Python script. In a disk image editing task, the human manually created device nodes with mknod to access partitions; the AI went straight to standard tools like fdisk, mkfs.ext4, and mount.
Different paths, same destination—which validates TerminalWorld's design philosophy: judge results, not processes. Tasks describe a final state, not how to reach it; any path to the correct end state counts.
Open-Source Models: 4-8x Cheaper, Comparable Performance
Another notable finding: open-source models are rapidly closing the gap with closed-source models.
On TerminalWorld-Verified, Kimi K2.6 and GLM 5.1 (57.5% and 57.0% pass rates) already exceed Gemini 3.1 Pro (55.0%) and GPT-5.5 (53.5%), at 1/4 to 1/8 of the cost:
| Model | Pass Rate | Cost per Solved Task | |-------|-----------|---------------------| | Claude Opus 4.7 | 62.5% | $0.51 | | Kimi K2.6 | 57.5% | $0.15 | | GLM 5.1 | 57.0% | $0.16 | | GPT-5.5 | 53.5% | $0.94 |
If your terminal tasks don't need that extra 5-10% pass rate, open-source models save 4-8x in cost.
Agent Frameworks: Lower Cost, No Higher Ceiling
The paper also tested 6 terminal agent frameworks (Terminus-2, Claude Code, mini-SWE-agent, OpenHands, Gemini CLI, Codex CLI). Key conclusion: agent frameworks mainly affect cost efficiency, not the capability ceiling.
With the same model (Claude Opus 4.7), pass rates ranged from 45% to 62.5% across frameworks, but costs differed enormously: Terminus-2 and mini-SWE-agent totaled about $60, while OpenHands reached $371.
Implication: good agent design should reduce exploration friction, not add orchestration complexity. Helping the model find the right solution path faster matters more than more tools or more elaborate pipelines.
Why TerminalWorld Matters
First, it exposes a blind spot in current evaluation. We've been measuring terminal capability with expert-designed "hard problems," yet TerminalWorld shows those scores are nearly uncorrelated with real performance. If your product relies on AI working at the terminal, TerminalWorld scores are more informative than Terminal-Bench.
Second, it offers a sustainable evaluation paradigm. Traditional benchmarks are one-shot—fixed once authored, and quickly outdated as tools and practices evolve. TerminalWorld's automated pipeline can continuously generate new tasks from new recordings.
Third, it reveals the real bottleneck for AI terminal assistants. It's not insufficient reasoning—it's inefficient exploration in open environments. Future breakthroughs may come not from bigger models but from smarter exploration strategies.
Fourth, the window of opportunity for open-source models. On real terminal tasks, the open/closed-source gap is already small while costs differ by 4-8x. For terminal automation, open-source models may be the more pragmatic choice.
---
Paper: https://arxiv.org/abs/2605.22535 Code: https://github.com/EuniAI/TerminalWorld Dataset: https://huggingface.co/datasets/EuniAI/TerminalWorld