When the Strongest AI Sits at the Terminal: 80,870 Real Screen Recordings Expose a Harsh Truth
Have you ever wondered: if you put Claude Opus 4.7—one of the strongest AI models available today—in front of a terminal to handle everyday command-line tasks a programmer deals with, how much could it actually get done?
The answer: less than two-thirds.
This isn't a joke. It's the core finding of TerminalWorld, a new paper from UCL (University College London) together with Nanjing University and Tencent. The researchers did something unprecedented: they collected 80,870 real terminal recordings of programmers from the internet, used an automated pipeline to turn them into 1,530 executable evaluation tasks, and then put 8 frontier LLMs and 6 terminal agents through the exam.
The result? The strongest performer, Claude Opus 4.7, achieved only a 62.5% pass rate. Meanwhile, the "top students" who score 80+ on expert-designed benchmarks fell flat on their faces when facing real-world tasks.
A Counter-Intuitive Finding: High Exam Scores ≠ Real-World Skill
First, an uncomfortable conclusion for many people.
The TerminalWorld paper compared its own results with Terminal-Bench, currently the most popular terminal benchmark, and found the correlation between them is only Pearson r = 0.20—which in statistical terms means roughly "no meaningful relationship."
What does this mean? GPT-5.5 scores a high 82.7% on Terminal-Bench, but on TerminalWorld's real tasks, it only manages 53.5%. Meanwhile the open-source model Kimi K2.6 scores just 66.7% on Terminal-Bench, but reaches 57.5% on TerminalWorld—outright beating GPT-5.5 and Gemini 3.1 Pro.
Why? Because expert-crafted questions and the real world are two different things.
When experts design benchmarks, they naturally lean toward "hard problems"—puzzles with clever solutions requiring deep reasoning. But in the real world, what programmers actually do in the terminal is: configure Kubernetes clusters, write CI/CD pipelines, debug Docker networking, provision infrastructure with Terraform... These tasks aren't necessarily "hard," but they require knowing the right tools and workflows, and finding the correct path within an open action space.
It's like a math olympiad gold medalist who is unbeatable on competition problems, but if you ask them to fix a pipe at home, they may not even know where the wrench is.
How Was TerminalWorld Built?
The most ingenious part of this paper isn't the evaluation results—it's the data construction method. Let me explain with an analogy.
Imagine you're organizing a cooking competition. The traditional approach is to invite master chefs to write the exam questions themselves—"make a molecular gastronomy dish"—refined but disconnected from daily life. TerminalWorld's approach is: find 80,870 real cooking videos on YouTube and reverse-engineer each one into an exam question.
Concretely, a four-step pipeline:
Step 1: Collect recordings. asciinema is a platform where programmers share terminal recordings—like a "game replay" service for programmers. The paper collected 80,870 recordings from there.
Step 2: Distill tasks. Real recordings are full of typos, repeated retries, and boring ls and cat commands. The paper used an LLM (Claude Sonnet 4.6) to extract two things from this noise: a "task specification" (describing only the goal, not the steps) and a "reference solution" (a clean command script). After filtering, 9,492 high-quality recordings remained.
Step 3: Reproduce environments. Recordings capture commands but not what was installed on the programmer's machine. The paper used Claude Code as the "environment engineer," inferring required dependencies from the reference solution, then writing Dockerfiles, building images, running containers, and replaying commands. Failures were repaired or discarded. Ultimately, 5,035 tasks had successfully reproduced environments.
Step 4: Generate tests. With tasks and environments in place, automated grading was still needed. The paper designed a "triple trial" mechanism:
- All-pass trial: run the reference solution; all tests must pass (prevents false negatives)
- No-op trial: run nothing; all tests must fail (prevents hollow tasks)
- Partial trial: run a broken solution; at least one test must fail (prevents overly lenient tests)
The pipeline's greatest strength: it's fully automated and can run continuously. New recordings are uploaded to asciinema every day, and TerminalWorld can periodically rerun the pipeline, forever keeping up with the latest programmer practices. This is what the paper calls "authentic and scalable by construction"—reliable because it's real; sustainable because it's automated.
The Efficiency Paradox: The Harder You Try, the More You Fail?
The most thought-provoking finding in the evaluation is what the paper calls the "Efficiency Paradox."
In TerminalWorld, task success rate is negatively correlated with resource consumption—the more tokens and turns used, the more likely the task is to fail. Specifically, failed tasks consume on average 3.3x more tokens than successful ones, accounting for 63% of total cost while representing only 43% of attempts.
GPT-5.5 and MiniMax M2.7 are the two most typical "grinders"—they consume far more tokens and turns than other models, yet their pass rates are lower.
Why? Because real terminal tasks present the AI with an open action space. Unlike coding problems with clear inputs and outputs, terminal tasks may require installing software, configuring environments, running commands, checking results... every step offers countless choices. Without reliable planning and stopping strategies, the AI keeps exploring the environment, going further and spending more, without getting closer to the right answer.
It's like a person in a maze who doesn't know where the exit is, so they keep trying every path. Walking more doesn't mean getting closer to the exit.
Do AI and Humans Take the Same Path?
TerminalWorld has a unique advantage: every task comes from a real human recording, so AI and human solution paths can be compared.
The finding: AI and humans almost never take the same path. The median command-set overlap is only 21.4%.
For example, in a network packet analysis task (extracting HTTP Basic Auth credentials from a pcap file), the human used ettercap to replay and parse the capture file, while the AI used tshark plus a Python script to parse it directly. In another task modifying a disk image, the human manually used mknod to create device nodes to access partitions, while the AI used standard tools like fdisk, mkfs.ext4, and mount directly.
Different roads, same destination—but completely different paths. This precisely validates TerminalWorld's design philosophy: judge the result, not the process. Tasks describe a "final state," not "how to get there." Any path that reaches the correct end state counts.
Open-Source Models: 4-8x Cheaper, Comparable Performance
Another notable finding: open-source models are rapidly closing the gap with closed models.
On TerminalWorld-Verified, Kimi K2.6 and GLM 5.1 have pass rates (57.5% and 57.0%) that already exceed Gemini 3.1 Pro (55.0%) and GPT-5.5 (53.5%)—at only 1/4 to 1/8 the cost of closed models:
| Model | Pass Rate | Cost per Success | |-------|-----------|------------------| | Claude Opus 4.7 | 62.5% | $0.51 | | Kimi K2.6 | 57.5% | $0.15 | | GLM 5.1 | 57.0% | $0.16 | | GPT-5.5 | 53.5% | $0.94 |
Open-source models dominate on cost-effectiveness. If your terminal tasks don't need that extra 5-10% pass rate, open-source models can save you 4 to 8 times the money.
Agent Frameworks: Lower Costs, But No Higher Ceiling
The paper also tested 6 terminal agent frameworks (Terminus-2, Claude Code, mini-SWE-agent, OpenHands, Gemini CLI, Codex CLI) and reached an important conclusion: agent frameworks mainly affect cost efficiency, not the capability ceiling.
With the same model (Claude Opus 4.7), pass rates varied between 45%-62.5% across frameworks, but costs differed dramatically: Terminus-2 and mini-SWE-agent cost roughly $60 total, while OpenHands ran as high as $371.
This means: good agent design should reduce exploration friction, not add orchestration complexity. Helping the model find the correct solution path faster matters more than giving it more tools and more elaborate pipelines.
Why Does TerminalWorld Matter?
Finally, why should every AI practitioner care about this paper?
First, it exposes a blind spot in current evaluation systems. We've been measuring AI's terminal capabilities with expert-designed "hard problems," but TerminalWorld proves these scores have almost no relation to real-world performance. If your product relies on AI working in the terminal, TerminalWorld's scores are more informative than Terminal-Bench's.
Second, it offers a sustainable evaluation paradigm. Traditional benchmarks are one-shot—once the questions are written, they're frozen, and quickly become obsolete as tools and practices evolve. TerminalWorld's automated pipeline can continuously generate new tasks from new recordings, always staying current.
Third, it reveals the real bottleneck of AI terminal assistants. It's not insufficient reasoning ability—it's low exploration efficiency in open environments. Future breakthroughs may come not from bigger models, but from smarter exploration strategies.
Fourth, an opportunity window for open-source models. On real terminal tasks, the gap between open and closed models is already small, while costs differ by 4-8x. For the terminal automation scenario, open-source models may be the more pragmatic choice.
---
Paper: https://arxiv.org/abs/2605.22535 Code repository: https://github.com/EuniAI/TerminalWorld Dataset: https://huggingface.co/datasets/EuniAI/TerminalWorld