Introduction
If you want to train a diver to survive the open ocean, would you keep them in a hotel's heated pool? AI agents face the same problem: they excel at curated, expert-designed benchmarks, but real developer environments are a jungle of leftover configs, conflicting dependencies, and mysterious legacy scripts.
In May 2026, researchers from University College London (UCL) and Nanjing University (with Tencent) published "TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks" on arXiv. Their approach: reverse-engineer a "cyber wilderness" for AI from more than 80,000 real human terminal session recordings.
Reverse-Engineering: Extracting "Pain" from 80,000 Recordings
The team targeted asciinema.org, which hosts thousands of real command-line sessions—grueling debugging, cloud deployments, and maddening compile failures. Their automated pipeline:
1. Harvest recordings: Collected 80,000+ raw sessions, stripped of private information. 2. Extract intent: Used frontier models (e.g., Claude Sonnet 4.6) to interpret messy sessions and infer what the human was trying to accomplish. 3. Clone environments: Automatically reconstructed matching Docker containers to reproduce the messy real-world environment. 4. Auto-generate tasks: Produced validation scripts to objectively evaluate whether AI agents can complete the same tasks.
This bypasses the "human expert filter," exposing agents to raw, unpolished developer challenges for the first time.
The Efficiency Paradox: AI Falls into the "Brute-Force Trap"
The most striking finding is the Efficiency Paradox. In traditional benchmarks, more thinking time and tokens usually mean closer to the answer. In real terminal environments, the opposite holds: success rate correlates negatively with compute consumed.
When an agent gets confused, it doesn't stop and re-plan. Like a hiker without a compass, it spins in place—repeatedly running ls, cat-ing the same config files, installing nonexistent dependencies. This brute-force searching doesn't solve the problem; it drowns the agent in error messages.
Even the most capable models currently pass only 62.5% of tasks in this wilderness.
Open Questions: The Dark "Discrimination Black Box"
1. Hallucination boundaries in intent extraction: Can the LLM reverse-engineering human recordings suffer "second-order hallucinations"? If the human's actions were themselves wrong or misleading, would the engine mistake them for sophisticated operations? The reliability ceiling of distilling truth from noise is unknown. 2. The side-effect cleanup problem: Terminal operations carry broad side effects (global environment variables, background processes). Docker-based reproduction may incompletely capture complex cross-container, cross-network interactions. 3. Missing stopping criteria: We don't yet know how to give agents an "intuition alarm" to recognize unproductive loops and cut their losses before sinking into the Efficiency Paradox.
Conclusion
Intelligence isn't about memorizing recipes—it's about handling a messy kitchen. TerminalWorld shows the true unevenness of the real world: an agent that aces benchmarks may struggle with a single real "Segmentation Fault" more than a three-month junior developer.
Next time you watch an AI write code fluently, ask: thrown into a real terminal full of errors and legacy code, can it still find its way home?
*True strength is born in the moment of solving real-world chaos.*
---
Paper metadata: arXiv ID 2605.23126 (May 2026) — Zhaoyang Chu, Jiarui Hu, et al. (UCL, Nanjing University, Tencent)