Inducing Task Models from Computer-Use Traces (arXiv 2608.20319)

Forum topic · 小凯 · 2026-08-22

Summary

This paper introduces Task Model Induction (TMI), a method for deriving symbolic, auditable, and reusable task models from naturally occurring computer-use traces—passively recorded screenshots and mouse/keyboard events. As computer-use agents enter real workplaces, organizations need models of how tasks are actually performed for learning, auditing, and reuse. Deriving such models is challenging because activity is observed only as low-level events and real work is multi-threaded with interleaved goals. TMI addresses this in two stages: (i) it discovers latent tasks in unconstrained traces, disentangling concurrent activities; and (ii) for each latent task it induces a task model pairing a hierarchical goal model based on recursive goal decomposition with a control-flow process model of how the work is executed. On controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 consistency and reconstructs 74.9% of observed execution steps. In external evaluation, skills derived from TMI task models improve held-out task accuracy by 30.0%. Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang (NLP).

Paper Overview

Field: NLP Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang arXiv: 2608.20319

Abstract

Natural computer-use traces (passively recorded screenshots and mouse/keyboard actions) are a valuable resource for deriving symbolic, auditable, and reusable models of everyday work. As computer-use agents enter real-world work, such models become increasingly important—agents need to learn how tasks are actually performed, and organizations need to audit and reuse this knowledge.

However, inducing task models is challenging because activity is observed only as low-level events, and real work is multi-threaded with interleaved goals. Existing methods assume a given task or a single workflow, producing step-level summaries rather than structured task models.

This paper proposes Task Model Induction (TMI), which:

1. Discovers latent tasks in unconstrained traces, disentangling concurrent activities; 2. Induces a task model for each latent task, pairing a hierarchical goal model (via recursive goal decomposition) with a control-flow process model of how the work is executed.

Results

On controlled human and agent trajectories, TMI recovers interleaved tasks with a 0.974 consistency rate.
It reconstructs 74.9% of observed execution steps.
In external evaluation, skills derived from TMI task models improve held-out task accuracy by 30.0%.

--- *Auto-collected on 2026-08-22*