Generative Tutorial: Live Contextualized Visual Instructions via AR and Generative AI

Forum topic · 小凯 · 2026-09-23

Summary

A paper by Muzhe Wu, Zuchen Li, Xu Wang, and Anhong Guo (arXiv:2609.24955) introduces Generative Tutorial, a framework for live visual instructions for physical tasks that depict intended outcomes and actions within the user's own environment. The authors first conducted a formative evaluation of state-of-the-art image and video generation models across 15 physical tasks, identifying failure modes and potential benefits. Based on these findings, they built an augmented reality prototype that proactively generates goal images and demonstration videos using observed workspace context and predicted visual outcomes of preceding actions. A 24-participant lab study showed higher task completion quality, stronger perceived workspace correspondence, and shorter intervals between step confirmations compared with pre-authored instructions. Qualitative findings highlight how contextual similarity affects trust, how generation errors affect comprehension, and how instruction delivery should adapt to user needs.

Paper Overview

Field: Machine Learning Authors: Muzhe Wu, Zuchen Li, Xu Wang, Anhong Guo Published: 2026-09-21 arXiv: 2609.24955

Abstract (full translation)

Visual instructions for physical tasks are typically authored in one context and followed in another, requiring users to translate demonstrated tools, materials, and spatial relationships into their own environment. We introduce Generative Tutorial, a conceptual framework for live visual instruction that depicts intended outcomes and actions within the user's environment and task flow.

A formative evaluation of state-of-the-art image and video generation models identifies failure modes and potential benefits across 15 physical tasks. Drawing on these findings, we build an augmented-reality prototype system that proactively generates goal images and demonstration videos using observed workspace context and predicted visual outcomes of preceding actions.

A 24-participant lab study found higher task completion quality, stronger perceived workspace correspondence, and shorter intervals between step confirmations compared with pre-authored instructions. Qualitative findings highlight how contextual similarity affects trust, how generation errors affect comprehension, and how instruction delivery should adapt to user needs, providing implications for future design.

Key Contributions

Conceptual framework: Live visual instructions generated in the user's own environment and task flow, rather than authored in a different context.
Formative evaluation: Assessment of state-of-the-art image and video generation models across 15 physical tasks, surfacing failure modes and benefits.
AR prototype: Proactively generates goal images and demonstration videos grounded in observed workspace context and predicted outcomes of prior actions.
User study (N=24): Demonstrated improvements in task completion quality, perceived workspace correspondence, and step confirmation intervals versus pre-authored instructions.