Action-Sketcher: Why Robots Now Sketch Before They Act

Forum topic · 小凯 · 2026-05-05

Summary

Researchers from Tsinghua University, Beijing Institute of Technology, and Xiaomi have introduced Action-Sketcher, a new framework for embodied AI that lets robots reason visually before executing long-horizon manipulation tasks. Described in the paper 'Thinking in Text and Images: Interleaved Vision-Language Reasoning Traces for Long-Horizon Robot Manipulation' (arXiv:2601.01618), the approach inserts an explicit sketching step between perception and action: instead of mapping instructions directly to motor commands like traditional VLA (vision-language-action) models, the robot generates interleaved text-and-image reasoning traces—such as drawing bounding boxes over target objects or arrows indicating motion paths. This solves three key problems: it disambiguates which object to manipulate, enables dynamic self-correction when the environment changes mid-task, and supports human-in-the-loop control, since users can correct errors by simply editing the sketch rather than modifying code. The framework marks a shift from opaque, black-box control toward interpretable visual-spatial reasoning, which the authors argue is a necessary step toward general-purpose robots capable of handling complex, multi-step tasks like cleaning and sorting.

Overview

Embodied AI just crossed a threshold. A new paper (arXiv:2601.01618) from Tsinghua University, Beijing Institute of Technology, and Xiaomi introduces Action-Sketcher, a framework where robots don't just understand language—they first draw an internal visual sketch before acting. This moves robots beyond blind, black-box operation into what the author calls "chef mode": plan first, then execute.

1. The Problem: Why Old Robots "Freeze"

Ask a robot to "clean the kitchen and sort the trash"—a long-horizon task with dozens of sub-actions—and traditional VLA (Vision-Language-Action) models behave like students who memorized without understanding. They push instructions and camera frames into a giant mathematical black box and output raw motor values.

Weakness: If someone kicks a chair mid-task or the trash bag tears, the internal math collapses. The robot spins in place or jerks mechanically—because it never understood *why* it was acting, only fitting probabilities.

2. The Fix: A "Magic Pen" for Machines

think it through, sketch it, then act.

Sketch

Example—"put the blue cup on the table into the tray":

The robot doesn't reach immediately. On the camera image, it interleaves text and drawn symbols:
Text: "First I need to locate the cup." → sketch: a red box around the blue cup.
Text: "Then move it to the tray." → sketch: an arrow from the cup to the tray's center.

This interleaved text-image reasoning chain is like a chef practice-slicing motions on the cutting board before cooking.

3. Why Sketching Is Revolutionary

Disambiguation

Dynamic self-healing

Human-in-the-loop control

4. Conclusion: Embodied AI's "Cursor Moment"

explicit visual-spatial reasoning is the necessary path to general-purpose robots

---

Paper Details

Title: Thinking in Text and Images: Interleaved Vision-Language Reasoning Traces for Long-Horizon Robot Manipulation
Authors: Wentao Yu, et al. (Tsinghua University, Beijing Institute of Technology, Xiaomi)
Identifier: arXiv:2601.01618 (January 2026, widely discussed since May)
Keywords: Action-Sketcher, VLA Models, Visual Reasoning, Human-Robot Collaboration, Long-Horizon Tasks

*Deep translation by Stratagem, first published on zhichai.net.*