> HKUDS/CLI-Anything's core argument: GUI Agents are a paradigm error. Rather than making AI imitate humans clicking a mouse, turn software into a form AI can naturally use — the command line.
GitHub: https://github.com/HKUDS/CLI-Anything Paper: https://arxiv.org/abs/2606.03854
An Awkward Status Quo
It's 2026, and the mainstream way AI agents operate software is still: screenshot → recognize UI elements → compute coordinates → simulate mouse clicks.
This paradigm has a name: GUI Agent. It has a fatal flaw — it makes AI imitate human perceptual limitations.
Humans see screens with eyes, so software is designed as visual interfaces. But AI doesn't need eyes. Making a system that can directly process structured data parse pixels, find button positions, and simulate clicks is like making someone who can do mental arithmetic use an abacus — not impossible, just wasteful.
The CLI-Anything paper pinpoints the problem precisely:
> Current GUI agents struggle with brittle pixel-level interactions, timing dependencies, and coordinate-based actions that break with interface changes. They force agents to emulate human perceptual limitations rather than leverage their computational strengths in structured data processing and programmatic control.
In other words: GUI Agents are brittle in three places — pixel-level interactions, timing dependencies, coordinate-based actions. One interface change and everything breaks. They force AI to imitate human perceptual limitations instead of leveraging AI's computational strengths.
Change the Layer
CLI-Anything's solution is not "a better GUI Agent" — it's changing layers entirely:
> Instead of forcing agents to navigate visual layouts, we create interfaces aligned with how agents naturally operate: through structured commands, explicit state representations, and deterministic feedback.
Not making AI better at reading screens, but giving AI the interface it's naturally good at: structured commands, explicit state, deterministic feedback.
Concretely: convert existing applications into "command-line harnesses" — preserving functionality while exposing machine-readable protocols.
This eliminates the core pain point of GUI Agents: lossy visual-to-computational translation. AI no longer needs to "look" at a screen and guess where to click; it directly invokes a command.
CLI-Hub: An App Store for Agents
CLI-Anything isn't just a paper — it has a complete ecosystem: CLI-Hub.
After pip install cli-anything-hub, cli-hub install <name> installs community-built CLI harnesses.
Software currently covered by CLI-Hub (extracted from the README's News section):
- ArcGIS Pro: GIS mapping, geoprocessing, feature editing
- Obsidian: note automation, persistent memory
- Joplin: notebooks, todos, tags, attachments, E2EE
- Rekordbox: SQLCipher write path for DJ software
- Calibre: e-book library management, search, conversion, export
- 3MF: 3D mesh inspection, hole repair, triangle properties
- MiniMax: chat/TTS workflows
- QGIS: full GIS/map authoring
- UniMol Tools: molecular modeling
- UEAtelier: self-extending workbench for Unreal Editor
- Shotcut: video rendering
- n8n: workflow automation
- Zoom: recording download
- Octopus RNA editing: don't change DNA, change the blueprint
- Slime mold externalized memory: don't use neurons, use slime
- Avian quantum magnetoreception: don't use a magnetometer, use radical pairs
- SOPHIA division of labor: don't unify processing, route by state
- EvoThink atomic reasoning: don't optimize the whole chain, segment the reasoning stream
- Möbius RoPE topological intervention: don't change frequencies, change topology
- Mantis shrimp phononic shields: don't resist hard, filter selectively
- Euclid-MCP reasoning outsourcing: don't train models to reason, outsource to Prolog
- ACE context engineering: don't pile up context, compress in segments
- Cordis reversible effects: don't hand-write undo, lift it to the type layer
- CLI-Anything: don't build a better GUI Agent, turn software into CLI
- Requires source code or API: closed-source software without source code (e.g., Photoshop, AutoCAD) is hard to auto-generate CLI harnesses for
- Limited CLI expressiveness: some highly visual operations (drag-based color grading, freeform deformation) are hard to express as commands
- Variable generation quality: CLIs from the 7-stage pipeline may need manual correction
- Every software UI update means agents must be retrained
- Every agent failure involves pixel-level debugging
- Agent reliability is forever capped by "visual recognition accuracy"
- GitHub: https://github.com/HKUDS/CLI-Anything
- Paper: https://arxiv.org/abs/2606.03854
- Website: https://clianything.cc
- Install:
pip install cli-anything-hub - Stars today: 100 (trending 2026-08-15)
- Highlight: 7-stage automatic CLI generation + CLI-Hub ecosystem
Each CLI is an "agent-ready" interface — AI calls it directly, no screen-reading required.
A Seven-Stage Automated Generation Pipeline
The most hardcore part of CLI-Anything is its automation pipeline. According to developersdigest's coverage, it runs 7 stages over a software codebase and automatically generates a tested CLI harness, including a REPL.
This means: in theory, any software can be converted into a CLI harness. You don't have to write it by hand — the pipeline automatically analyzes the codebase, extracts functionality, generates commands, and writes tests.
This is the exact opposite of the GUI Agent approach. GUI Agents "adapt AI to existing interfaces"; CLI-Anything "adapts interfaces to AI."
The Latest Member of a Conceptual Lineage
CLI-Anything is another clear case in the "solve by changing layers" lineage:
The common pattern across all twelve cases: don't do the same thing harder — change layers so the problem disappears.
CLI-Anything's layer change: move the question of "how AI operates software" from the "visual adaptation" layer to the "interface redesign" layer. Once the layer changes, pixel recognition, coordinate computation, and timing dependencies all vanish — not solved, but bypassed.
Structural Isomorphism with Euclid-MCP
CLI-Anything and Euclid-MCP (studied previously) are structurally isomorphic:
Euclid-MCP: A 480B model is as bad as an 8B model on 1000-fact RBAC reasoning. Semantic retrieval executing formal rules = hammering nails with a power drill. Solution: outsource reasoning to Prolog.
CLI-Anything: GUI Agents are fragile when operating software. Visual interfaces executing programmatic operations = using an abacus with your eyes. Solution: convert interfaces into CLIs.
Shared pattern: acknowledge AI is unreliable on certain tasks, route around it, don't grind training on it.
Euclid-MCP admits LLMs are unreliable at formal reasoning and outsources reasoning to Prolog. CLI-Anything admits LLMs are unreliable at visual operation and outsources operation to a CLI.
It's another instance of the "division of labor beats unification" principle — don't pursue one model doing everything; let specialized tools do specialized jobs.
An Honest Boundary
CLI-Anything is not omnipotent. Its limitations:
But these are engineering problems, not paradigm problems. CLI-Anything's core thesis — "GUI Agents are a paradigm error" — holds.
Why It Matters Now
The Agent era is arriving. If the mainstream agent interaction model is GUI, we sink into a quagmire:
CLI-Anything offers another path: make software adapt to agents, not agents adapt to software.
This path needs an ecosystem — every piece of software needs a CLI harness. CLI-Hub is doing exactly that, and it already spans everything from GIS to e-books to DJ software.
If this ecosystem matures, the future of agents operating software will be as reliable as programmers calling APIs — not "looks like it clicked the right thing," but "it actually executed."
---
Project Info