Task Model Induction (TMI): Turning Screen Recordings into Auditable Task Knowledge

Forum topic · ✨步子哥 · 2026-08-21

Summary

Researchers from Stanford and CMU propose TMI (Task Model Induction), a framework that automatically derives symbolic, hierarchical task models from passively recorded computer-use traces. TMI addresses three core challenges: grounding low-level events (clicks, keystrokes, screenshots) into semantic actions, separating interleaved multitasking within a single recording, and capturing hierarchical goal structures. The three-stage pipeline performs event grounding and activity segmentation, latent task induction (achieving 0.974 agreement with ground-truth labels under heavy interleaving), and task model construction combining an Objective Model (recursive goal decomposition tree) with a Procedure Model (control flow with loops and conditionals). On controlled human and agent trajectories, TMI reaches 74.9% step-description accuracy versus 30.3% for the strongest baseline, 88.5% operator accuracy versus 52.7%, and transferring induced task models as skills improves held-out task accuracy by 30%. Beyond agent learning, the work enables organizational auditing of how work is actually done and automatic training material generation, converting tacit work knowledge into explicit, auditable, transferable knowledge.

You record a full day of screen activity—switching windows, filling forms, checking email, replying to messages, going back to the form. Hidden in these actions is your workflow: what tasks you're doing, the steps of each task, and how you switch between them.

If AI could automatically extract a workflow model from such a recording—not a low-level log of "which button you clicked," but structured knowledge of "what goals you're pursuing and how each goal decomposes"—it could learn how to help you work.

That's what this paper by Yucheng Jiang et al. from Stanford and CMU does. Their TMI (Task Model Induction) framework automatically induces symbolic task models from passively recorded computer-use traces.

Why It's Hard

The problem has three core challenges:

First, you see events, not tasks. Screen recordings give you screenshots plus mouse/keyboard actions—"clicked (847, 320)", "pressed Ctrl+C", "switched to window 3". These events don't tell you what the user is doing. You must first ground low-level events into semantic actions ("copied data from the spreadsheet"), then group actions into tasks ("filling in the quarterly report").

Second, real work is interleaved. People don't finish one thing before starting the next. You fill in two rows of a form, switch to email to reply, return to the form, and look something up in between. This interleaving breaks the naive "one recording, one task" assumption.

Third, task structure is hierarchical. "Write the quarterly report" contains sub-goals like "collect data", "make charts", "write analysis", each with concrete steps. A task model needs to capture this hierarchy, not just a linear step list.

TMI's Three-Stage Pipeline

1. Event grounding and activity segmentation: A vision-language model grounds the low-level screenshot + input event stream into semantic actions ("entered a value in Excel"), then segments them into activity episodes by semantic continuity.

2. Latent task induction: Large language models cluster interleaved activity episodes into latent tasks. The key is handling interleaving—three tasks may alternate within one recording, and TMI must separate them. The paper reports task-induction stability: under heavy interleaving, agreement with ground-truth annotations reaches 0.974.

3. Task model construction: For each task, TMI builds a hierarchical model with two parts:

Objective Model: a recursive goal decomposition tree. "Write quarterly report" → "collect data" + "make charts" + "write analysis"
Procedure Model: a control-flow description—loops, conditional branches, sub-step ordering

The two models are complementary—the Objective Model says "what to do," the Procedure Model says "how." The paper finds either one alone is insufficient: goal models alone lose procedural detail; procedure models alone lose goal structure. They are merged via a "model reconciliation" step for complete coverage.

The Numbers

Tested on controlled human and agent trajectories:

Step-description accuracy 74.9%, vs. 30.3% for the strongest baseline—more than double
Operator accuracy 88.5%, baseline 52.7%
Task-induction agreement 0.974 (vs. ground-truth annotations)
External validation: using TMI's induced task models as skills transferred to new tasks improves held-out task accuracy by 30.0%

These numbers show TMI isn't just "reconstructing" what you did—it's "understanding" why. The extracted task models are transferable, reusable knowledge.

Why It Matters

The deeper value goes beyond technique—it points to a new knowledge-acquisition paradigm.

Traditional AI training relies on annotated data: humans write "step 1, step 2, step 3" and models learn. But much work knowledge is tacit—you can't even articulate what you do daily and why. TMI extracts task models from passive recordings, converting "tacit work knowledge" into "explicit, auditable knowledge."

Three scenarios benefit directly:

1. Agent learning: AI agents no longer need hands-on teaching—they can learn from recordings of your work 2. Organizational auditing: companies can audit how work is *actually* done, versus how process documents claim it's done—these often differ substantially 3. Skill transfer: veteran employees' recordings automatically become training material for new hires

More broadly, TMI embodies a "granularity alignment" principle: it lifts task modeling from the granularity of individual events to a goal–procedure hierarchical structure that matches real work. Prior approaches were either too fine (event-level pipeline decomposition) or too coarse (single-task workflows). TMI's hierarchical model is the first to align granularity with reality.

---

Paper: Inducing Task Models from Computer-Use Traces

Code: GitHub: Yucheng-Jiang/task-model-induction

Authors: Yucheng Jiang, Zora Zhiruo Wang (CMU), Ruishi Chen, Diyi Yang (Stanford)