TMI: Inducing Auditable Task Models from Passive Screen Recordings

Forum topic · ✨步子哥 · 2026-08-21

Summary

Researchers from Stanford and CMU propose TMI (Task Model Induction), a framework that automatically induces symbolic, hierarchical task models from passively recorded computer-use traces. The method addresses three challenges: grounding low-level events (screenshots, mouse/keyboard actions) into semantic actions, separating interleaved multitasking, and capturing hierarchical goal structure. TMI's three-stage pipeline performs event grounding and activity segmentation, latent task induction, and task model construction with two complementary components: an Objective Model (recursive goal decomposition tree) and a Procedure Model (control flow with loops and conditionals), merged via model reconciliation. On controlled human and agent trajectories, TMI achieves 74.9% step description accuracy versus 30.3% for the strongest baseline, 88.5% operator correctness, 0.974 task induction consistency under heavy interleaving, and a 30.0% accuracy gain when its induced models are transferred to held-out tasks. Beyond agent learning, TMI turns implicit work knowledge into explicit, auditable knowledge useful for organizational auditing and skill transfer. Paper: arXiv:2608.20319.

You record a full day of screen activity — switching windows, filling forms, checking email, replying to messages, going back to the form. Hidden in these traces is your workflow: what tasks you're doing, the steps of each task, and how you switch between them.

If AI could automatically extract a workflow model from such a recording — not a low-level log of which buttons you clicked, but structured knowledge of what goals you're pursuing and how each goal decomposes — it could learn how to help you work.

That is the contribution of this paper by Yucheng Jiang et al. from Stanford and CMU. Their TMI (Task Model Induction) framework automatically induces symbolic task models from passively recorded computer-use traces.

Why It's Hard

Three core challenges:

1. You see events, not tasks. Screen recordings give a stream of screenshots and mouse/keyboard actions — "clicked (847, 320)", "pressed Ctrl+C", "switched to window 3". These events alone don't reveal the user's task. The system must first ground low-level events into semantic actions ("copied data from the spreadsheet"), then group actions into tasks ("filling in the quarterly report").

2. Real work is interleaved. People rarely finish one task before starting another: fill two rows of a form, jump to email, come back, look something up in between. This interleaving breaks any assumption that one recording segment maps to one task.

3. Task structure is hierarchical. "Write quarterly report" decomposes into sub-goals like "collect data", "make charts", "write analysis", each with concrete steps. Task models must capture this hierarchy, not just a flat list of steps.

TMI's Three-Stage Pipeline

1. Event grounding and activity segmentation: Use vision-language models to ground sequences of screenshots + mouse/keyboard events into semantic actions ("entered a value in Excel"), then segment them into activity episodes based on semantic continuity.

2. Latent task induction: Use large language models to cluster interleaved activity episodes into latent tasks. The key is handling interleaving — the same recording may contain 3 tasks alternating. Under heavy interleaving, induced tasks reach 0.974 consistency with ground-truth annotations.

3. Task model construction: For each task, build a hierarchical model with two parts:

Objective Model: a recursive goal decomposition tree. "Write quarterly report" → "collect data" + "make charts" + "write analysis"
Procedure Model: control-flow description — loops, conditional branches, sub-step ordering

These two models are complementary: the Objective Model tells you *what* to do, the Procedure Model tells you *how*. The paper shows that neither alone is sufficient — objectives alone lose procedural detail; procedures alone lose goal structure. A model reconciliation step merges them for full coverage.

The Numbers

On controlled human and agent trajectories:

Step description accuracy: 74.9% vs. 30.3% for the strongest baseline — more than double
Operator correctness: 88.5% vs. baseline 52.7%
Task induction consistency: 0.974 against ground-truth annotations
External validation: transferring TMI's induced task models as skills to new tasks yields a 30.0% accuracy gain on held-out tasks

These results suggest TMI doesn't just reconstruct what you did — it understands why, producing transferable, reusable knowledge.

Why It Matters

The deeper value points to a new paradigm for knowledge acquisition.

Traditional AI training relies on labeled data: humans write "step 1, step 2, step 3" and models learn. But much work knowledge is tacit — you couldn't articulate exactly what you do daily or why. TMI extracts task models from passive recordings, effectively converting "implicit work knowledge" into "explicit, auditable knowledge."

This has direct value in three scenarios:

1. Agent learning: AI agents can learn from your work recordings instead of being hand-taught 2. Organizational auditing: companies can audit how work is *actually* done, not how process documents claim it's done — often very different 3. Skill transfer: veteran employees' recordings become automatically generated training material for newcomers

More broadly, TMI embodies a "granularity alignment" principle: it lifts task modeling from the granularity of individual events to a hierarchical goal-procedure structure that matches real work. Prior approaches either decompose event-level pipelines (too fine-grained) or model single-task workflows (too coarse). TMI's hierarchical model is the first to align with the true granularity of work.

---

Paper: Inducing Task Models from Computer-Use Traces

Code: GitHub: Yucheng-Jiang/task-model-induction

Authors: Yucheng Jiang, Zora Zhiruo Wang (CMU), Ruishi Chen, Diyi Yang (Stanford)