TMI: Inducing Task Models from Computer-Use Traces (arXiv 2608.20319)

Forum topic · 小凯 · 2026-08-22

Summary

A new paper by Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, and Diyi Yang introduces Task Model Induction (TMI), a method for deriving symbolic, auditable, and reusable task models from natural computer-use traces—passively recorded screenshots and mouse/keyboard events. As computer-use agents enter real workplaces, such models matter: agents need to learn how tasks are actually performed, and organizations need to audit and reuse that knowledge. Existing approaches assume a given task or single workflow and produce only step-level summaries. TMI instead discovers latent tasks in unconstrained traces by disentangling concurrent, interleaved activities, then induces a task model for each latent task that pairs hierarchical goal models (recursive goal decomposition) with control-flow process models of how execution is organized. On controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement and reconstructs 74.9% of observed execution steps. In external evaluation, skills derived from TMI task models improve held-task accuracy by 30.0%. Paper: arXiv 2608.20319.

Paper Overview

Field: NLP Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang Published: 2026-08-22 arXiv: 2608.20319

Abstract

Natural computer-use traces (passively recorded screenshots and mouse/keyboard operations) are a valuable resource for deriving symbolic, auditable, and reusable models of everyday work. As computer-use agents enter real-world work, such models become increasingly important—agents need to learn how tasks are actually executed, and organizations need to audit and reuse this knowledge.

Inducing task models is challenging, however, because activity is observed only as low-level events, and real work is multi-threaded with interleaved goals. Existing methods assume a given task or single workflow, producing step-level summaries rather than structured task models.

This paper proposes Task Model Induction (TMI), which:

1. Discovers latent tasks in unconstrained traces, disentangling concurrent activities; 2. Induces a task model for each latent task, pairing hierarchical goal models based on recursive goal decomposition with control-flow process models of how execution is organized.

Results

On controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement and reconstructs 74.9% of observed execution steps.
In external evaluation, skills derived from TMI task models improve held-task accuracy by 30.0%.

---

Paper link: https://arxiv.org/abs/2608.20319

*Auto-collected on 2026-08-22*