Computer-Use Agents Are Embodied Generalists
- Field: Machine Learning
- Authors: Pranjal Aggarwal, Sean Welleck
- Published: 2026-11-06
- arXiv: 2611.09847
- In many areas of AI, general-purpose models have replaced handcrafted methods and specialized architectures. The authors argue embodied AI is poised for a similar shift.
- Core claim: embodied tasks can be formulated as interaction with digital software.
- The authors build software that renders an agent's view of a physical environment on a screen and provides controls for acting within it.
- A computer-use agent (CUA) can then perform embodied tasks using the same screenshots, keyboard, and mouse actions it uses to operate ordinary software.
- Using a single CUA design and changing only the software, their entries won first-, second-, third-, and sixth-place prizes across four ECCV workshop competitions — with no competition-specific agent tuning.
- Skills CUA models learn on software, such as visual grounding, precise clicking, and exploration, appear to transfer to embodied tasks.
- Conclusion: computer-use agents can act as embodied generalists; with training strategies that make CUAs general, embodied AI may converge toward a single universal model, opening a new era of general-purpose embodied AI.
Key points
Original abstract (excerpt)
> In many areas of AI, general-purpose models have increasingly replaced handcrafted methods and specialized architectures. We argue that embodied AI is poised for a similar shift. Our key claim is that embodied tasks can be formulated as interaction with digital software. Specifically, we build software that renders an agent's view of a physical environment on a screen and provides controls for acting in it. A computer-use agent (CUA) can then perform embodied tasks through the same screenshots and keyboard and mouse actions that it uses to operate ordinary software. Using a single CUA design and changing only the software, our entries win first-, second-, third-, and sixth-place prizes across four ECCV workshop competitions, without any competition-specific agent tuning...
*Auto-collected on 2026-11-10.*