Summary
A zhichai.net forum post introduces the paper 'Inducing Task Models from Computer-Use Traces' (arXiv:2608.20319) by Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, and Diyi Yang, in NLP. The work explores deriving symbolic, auditable, and reusable models of everyday work from naturalistic computer-use traces—passively recorded screenshots and mouse or keyboard actions. As computer-use agents enter real workplaces, such models matter because agents need to learn how tasks are actually performed, while organizations need to audit and reuse that knowledge. The challenge is that activities are observed only as low-level events, whereas real-world workflows involve heavy branching, error recovery, and implicit contextual dependencies. The proposed method abstracts low-level events into high-level actions, identifies recurring patterns, and builds hierarchical task representations. Evaluated on a large dataset of office workers, the approach recovers interpretable, reusable workflow models that can serve as a foundation for automation and auditing.
Paper Overview
Field: NLP
Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang
Published: 2026-08-22
arXiv: 2608.20319
Summary
Naturalistic computer-use traces—passively recorded screenshots and mouse or keyboard actions—are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. As computer-use agents enter real-world work, these models become increasingly important: agents need to learn how tasks are actually performed, and organizations need to audit and reuse this knowledge.
Inducing such task models is challenging, however, because activities are observed only as low-level events, while real-world workflows involve substantial branching, error recovery, and implicit contextual dependencies.
The paper proposes a method for inducing task models from computer-use traces that:
- Abstracts low-level events into high-level actions
- Identifies recurring patterns
- Builds hierarchical task representations
The method is evaluated on a large dataset of office workers, demonstrating that it can recover interpretable and reusable workflow models, providing a foundation for automation and auditing.
Original Abstract (excerpt)
> Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done.
---
*Auto-collected on 2026-08-24*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178633921