Paper: Inducing Task Models from Computer-Use Traces

Forum topic · 小凯 · 2026-08-24

Summary

A forum post on zhichai.net introduces the arXiv paper 'Inducing Task Models from Computer-Use Traces' (arXiv:2608.20319) by Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, and Diyi Yang in the NLP field. The paper proposes a method for deriving symbolic, auditable, and reusable task models from naturalistic computer-use traces—passively recorded screenshots and mouse or keyboard actions. As computer-use agents enter real workplaces, such models become important: agents need to learn how tasks are actually performed, and organizations need to audit and reuse this knowledge. The key challenge is that activities are observed only as low-level events, while real-world workflows involve branching, error recovery, and implicit contextual dependencies. The proposed approach abstracts low-level events into high-level operations, identifies repeated patterns, and builds hierarchical task representations. Evaluated on a large dataset of office workers, the method recovers interpretable and reusable workflow models that can serve as a foundation for automation and auditing.

Paper Overview

Field: NLP

Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang

Published: 2026-08-22

arXiv: 2608.20319

Summary (translated)

Naturalistic computer-use traces—passively recorded screenshots and mouse or keyboard actions—are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. As computer-use agents enter real workplaces, these models become increasingly important: agents need to learn how tasks are actually performed, while organizations need to audit and reuse this knowledge.

However, inducing such task models is challenging because activities are observed only as low-level events, while real-world workflows involve extensive branching, error recovery, and implicit contextual dependencies. This paper proposes a method for inducing task models from computer-use traces that:

Abstracts low-level events into high-level operations
Identifies repeated patterns
Builds hierarchical task representations

The method is evaluated on a large dataset of office workers, demonstrating that it can recover interpretable and reusable workflow models, providing a foundation for automation and auditing.

Original Abstract (excerpt)

> Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done.

--- *Auto-collected on 2026-08-24*