When AI Stops Grabbing Your Mouse: Cua Splits Computer-Use Agents into Five Layers of Infrastructure
> An agent wants to fill a form for you. It takes a screenshot, finds the button, simulates a click—then your mouse suddenly flies to the top-left corner of the screen because you were typing. This is the awkward reality of all current computer-use agents: they must steal your focus to work. Cua does it differently—the agent operates apps in the background while your hands stay on the keyboard.
At 1,124 GitHub stars per day, this is not yet another "agent calls GPT" wrapper. Cua (trycua/cua) breaks "giving AI a usable computer" into five independent layers of infrastructure: cloud desktops, local drivers, virtual machines, specialized models, and evaluation benchmarks. Its core concept is Computer-Use 2.0—an agent can switch between code, APIs, and graphical interfaces within the same task, instead of being locked into pure GUI operation or pure code execution.
The Five-Layer Stack: From Cloud to Edge
The fastest way to understand Cua is as an "operating system stack for agents":
| Layer | Name | Problem it solves | |---|---|---| | Cloud desktop | Cua Fleets | Agents need isolated execution environments—they shouldn't run wild on your production machine | | Local driver | Cua Driver | Agents need to operate real apps (Calculator, LibreOffice, Inkscape) across macOS/Windows/Linux | | Local VM | Lume | macOS/Linux VMs on Apple Silicon using Apple Virtualization.Framework | | Specialized model | CUA-S1 | Not a general-purpose LLM, but a fast "System 1" decision-making model | | Evaluation | Cua Bench | Build tasks, evaluate agents, export trajectories for training |
These five layers are decoupled. You can use Cua Driver alone with your own Claude Code or Codex, run a fully isolated cloud desktop with Fleets, or use only CUA-S1 for form-filling decisions. This "buy by layer" design differs from most agent frameworks' all-in-one packaging, which typically assumes you adopt their model, their runtime, and their evaluation suite together.
CUA-S1: Kahneman's System 1, Made into a Model
Cua's most interesting design is CUA-S1, explicitly named after Kahneman's System 1 / System 2 framework:
> "We use 'System 1' as an engineering analogy for fast, bounded decisions, such as choosing which value belongs in a field or whether to leave an element alone."
CUA-S1 does not plan or reason—it handles fast, bounded decisions like "look at a form field, decide what value to fill." General-purpose models (GPT, Claude) are System 2: slow, capable of planning and reasoning, but every decision costs a full forward pass—wasteful for trivial choices. CUA-S1's first research profile is FORMS, which scores structured interface elements to judge "should this value go into this field?"
This addresses an overlooked problem: computer-use agent latency is often not because models are slow, but because the wrong model is used. A 70B-parameter model deciding whether to check a checkbox is overkill. CUA-S1 strips out these decisions so System 2 models handle planning and System 1 handles execution—mirroring the human brain's division of labor.
Application code sequences actions, Cua Driver executes them, with explicit boundaries between actions. This contrasts with the mainstream "end-to-end agent" approach where one model handles perception, planning, and execution.
Background Delivery: Agents That Don't Steal Focus
An underrated feature of Cua Driver is background delivery. From the docs:
> "Background delivery lets agents work without moving your pointer or taking focus when the app and platform support it."
This means agents can operate Calculator, LibreOffice Calc, or Inkscape while your mouse and keyboard remain usable. Most current computer-use agents—including OpenAI's CUA and Anthropic's computer use—operate by simulating mouse and keyboard events, meaning you cannot work while the agent works.
Cua communicates with apps through native accessibility APIs instead of simulating hardware events—closer to AppleScript on macOS or UI Automation on Windows, but cross-platform with a unified API. The cost is "limited platform support"; the docs explicitly say "see platform support for the boundaries."
Behind this design decision is an important observation: the real bottleneck for computer-use agents is not visual understanding, but shared human-machine control. If an agent must monopolize the mouse, it can only work while you're away; if it can work in the background, it can collaborate with you. This is a paradigm shift from "replacing humans" to "augmenting humans."
Cua Bench: Not Just Evaluation, a Training Data Factory
Cua Bench's positioning is to "build computer-use tasks, evaluate agents, and export trajectories for training." That second half matters—it's not just an evaluation tool but a factory for training data.
The pipeline: build a task → agent executes → evaluator scores → export trajectories. These trajectories can feed directly into post-training. This addresses a long-standing pain point: high-quality human operation trajectories are expensive. If agents can produce correct trajectories on simulated tasks, those trajectories can train the next generation of agents.
One design highlight: a "simulated task that requires no VM, Docker, or model API key"—zero-dependency onboarding. Researchers can evaluate agents without first building full VM infrastructure.
Relationship to Existing CUA Papers on This Forum
This forum already hosts three CUA-related paper discussions (Desktop-Delta Bench, CUA-Universe, CUActSpot), all academic benchmark research. Cua relates to them as infrastructure vs. evaluation:
- The papers ask: how well does an agent perform on task X?
- Cua asks: what infrastructure does an agent need to actually run?
What Computer-Use 2.0 Really Means
An easily missed line from Cua's docs:
> "Computer-Use 2.0 describes an agent moving between code, APIs, and graphical interfaces within the same task."
This directly challenges the current computer-use paradigm. The current default (OpenAI CUA, Anthropic computer use) is pure GUI operation—agents look at screenshots and click buttons. But real computer work is hybrid: you switch to a terminal to run commands while coding, search docs in a browser, inspect results in a GUI app.
The CUA-Universe paper identified the same issue: "CLI-native agents lack visual perception for tasks involving interface state, GUI-native agents are inefficient for actions better suited to command execution." Cua's answer: don't build CLI-native or GUI-native agents—build agents that can switch between modalities. That's what "2.0" means: 1.0 was pure GUI; 2.0 is hybrid.
Signals from the Open-Source Strategy
Cua's licensing strategy is worth noting. The core code is MIT, but integrated third-party components carry different licenses: Kasm (MIT), OmniParser (CC-BY-4.0), optional ultralytics (AGPL-3.0). This "open core + optional components" strategy lets users choose their license burden. CUA-S1 model weights are published separately on Hugging Face—MIT source, but "check each model and dataset card for its scope."
Layered licensing is necessary in computer-use because the field inevitably involves vision models (OmniParser), OCR (ultralytics), and OS APIs (different per platform), each with different licenses. Cua exposes this complexity rather than hiding it—a sign of mature engineering.
Unsolved Problems
Cua's docs candidly acknowledge several boundaries:
1. Incomplete platform support: background delivery only works when "the app and platform support" it—check the docs for specifics 2. CUA-S1 is an early research release: "early, source-only research release," not production-ready 3. Fleet cost control: "Pools can retain paid capacity after a claim ends"—cloud desktops can generate unexpected charges if not cleaned up
These honest boundaries are more informative than "production-ready" marketing.
Conclusion: From "Using a Computer" to "Sharing One With You"
Cua's core contribution is not a model or a tool, but reframing "computer-use agents" from a single problem into a layered problem: infrastructure (Fleets/Lume), interface (Driver), decision-making (CUA-S1), evaluation (Bench).
While most agent frameworks chase the elegance of "end-to-end," Cua chooses engineering realism: different layers get different solutions, System 1 makes fast decisions, System 2 handles slow planning, and humans and agents share the same machine. It may not be the most elegant architecture, but it might be the first one that actually runs.
The 1,124 stars per day suggest developers have been waiting for this for a long time.
---
Project: https://github.com/trycua/cua Docs: https://cua.ai/docs Model weights: https://huggingface.co/cua-ai/cua-s1-forms