Task Model Induction: Learning Structured Task Models from Computer-Use Traces

Forum topic · 小凯 · 2026-08-22

Summary

This paper introduces Task Model Induction (TMI), a method for deriving symbolic, auditable, and reusable task models from natural computer-use traces—passively recorded screenshots and mouse/keyboard events. As computer-use agents enter real work settings, organizations need models that capture how tasks are actually performed, both for agent learning and for auditing. Deriving such models is challenging because activity appears only as low-level events, and real work is multi-threaded with interleaved goals. TMI addresses this in two stages: (i) it discovers latent tasks in unconstrained traces and disentangles concurrent activities, and (ii) for each latent task it induces a task model pairing a hierarchical goal model (via recursive goal decomposition) with a control-flow process model of how the work is executed. On controlled human and agent trajectories, TMI recovers interleaved tasks with a 0.974 consistency rate and reconstructs 74.9% of observed execution steps. In extrinsic evaluation, skills derived from TMI task models improve held-out task accuracy by 30.0%. The work is by Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, and Diyi Yang, available on arXiv (2608.20319).

Paper Overview

Research Area: NLP Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang Published: 2026-08-22 arXiv: 2608.20319

Abstract

Natural computer-use traces—passively recorded screenshots and mouse/keyboard actions—are a valuable resource for deriving symbolic, auditable, and reusable models of everyday work. As computer-use agents enter real workplaces, such models become increasingly important: agents need to learn how tasks are actually performed, and organizations need to audit and reuse this knowledge.

However, inducing task models is challenging because activity is observed only as low-level events, and real work is multi-threaded with interleaved goals. Existing approaches assume a given task or a single workflow, producing step-level summaries rather than structured task models.

The TMI Approach

This paper proposes Task Model Induction (TMI), which:

1. Discovers latent tasks in unconstrained traces and disentangles concurrent activities. 2. Induces a task model for each latent task, pairing a hierarchical goal model (via recursive goal decomposition) with a control-flow process model of how the work is executed.

Results

On controlled human and agent trajectories, TMI recovers interleaved tasks with a 0.974 consistency rate.
It reconstructs 74.9% of observed execution steps.
In extrinsic evaluation, skills derived from TMI task models improve held-out task accuracy by 30.0%.

--- *Auto-collected on 2026-08-22*