Inducing Task Models from Computer-Use Traces: The TMI Framework (arXiv 2608.20319)

Forum topic · 小凯 · 2026-08-22

Summary

A new paper, Inducing Task Models from Computer-Use Traces (arXiv 2608.20319), introduces Task Model Induction (TMI), a method for deriving symbolic, auditable, and reusable task models from naturally collected computer-use traces such as passively recorded screenshots and mouse/keyboard events. As computer-use agents enter real workplaces, organizations need models of how tasks are actually performed so agents can learn from practice and workflows can be audited and reused. The challenge is that activities are observed only as low-level events, while real work is multi-threaded with interleaved goals. TMI addresses this by (i) discovering latent tasks in unconstrained traces and disentangling concurrent activities, and (ii) deriving a task model for each latent task, pairing hierarchical goal models with recursive goal decomposition and control-flow process models. On controlled human and agent trajectories, TMI recovers interleaved tasks with a 0.974 agreement rate and reconstructs 74.9% of observed execution steps. In extrinsic evaluation, skills derived from TMI task models improved holdout task accuracy by 30.0%.

Paper Overview

Field: NLP Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang Published: 2026-08-22 arXiv: 2608.20319

Abstract

Natural computer-use traces — passively recorded screenshots and mouse/keyboard operations — are a valuable resource for deriving symbolic, auditable, and reusable models of everyday work. As computer-use agents enter real-world work, such models become increasingly important: agents need to learn how tasks are actually performed, and organizations need to audit and reuse this knowledge.

However, inducing task models is challenging because activity is observed only as low-level events, and real work is multi-threaded with interleaved goals. Existing approaches assume a given task or a single workflow, producing step-level summaries rather than structured task models.

The Proposed Method: Task Model Induction (TMI)

This paper proposes Task Model Induction (TMI), which:

1. Discovers latent tasks in unconstrained traces and disentangles concurrent activities. 2. Derives a task model for each latent task, pairing a hierarchical goal model (with recursive goal decomposition) with a control-flow process model of how execution is organized.

Results

On controlled human and agent trajectories:

TMI recovers interleaved tasks with a 0.974 agreement rate.
It reconstructs 74.9% of observed execution steps.
In extrinsic evaluation, skills derived from TMI task models improve holdout task accuracy by 30.0%.