What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

Forum topic · 小凯 · 2026-09-09

Summary

A paper by Vivek Chavan, Pengtao Xie, Yahuan Shi, Oliver Heimann, Kevin Haninger, and Jörg Krüger (arXiv:2609.05376) investigates why visuomotor imitation policies that perform well in-distribution fail when visually similar objects or receptacles appear. Framing this as conditional visual grounding—where the visual target changes with manipulation phase and task state—the authors use Action Chunking with Transformers (ACT) with controlled color/shape distractors and localize failures to picking and placement phases. They find distractor sensitivity is specific to visual similarity type and manipulation stage, and evaluate distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting as complementary interventions, significantly improving robustness in simulation and on a physical UR3e robot. The same failure modes appear in pretrained vision-language-action policies, and explicit grounding improvements substantially recover performance.

Paper Overview

Research Area: Computer Vision (CV) Authors: Vivek Chavan, Pengtao Xie, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger Published: 2026-09-04 arXiv: 2609.05376

Abstract

Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. This paper studies the behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state.

Methodology

Uses Action Chunking with Transformers (ACT) as the base policy
Systematically introduces distractor objects and receptacles with controlled color and shape similarity
Localizes failures to the picking and placement phases

Key Findings

Distractor sensitivity is specific to both the type of visual similarity and the manipulation stage
Guided by this diagnosis, the authors evaluate three complementary interventions:

1. Distractor augmentation 2. Phase-dependent attention regularization 3. Appearance-based visual prompting

These interventions improve target selection while preserving the spatial information required for control, significantly improving robustness in simulation and on a physical UR3e robot.

Extension to VLA Policies

The same failure modes are examined in pretrained vision-language-action (VLA) policies. For example, the observed state of a medical instrument determines the correct destination. Results show that even when underlying manipulation skills are intact, visual distractors can cause incorrect object or destination selection—and explicitly improving target selection substantially recovers performance across different visuomotor policy learning paradigms.

--- *Source: arXiv:2609.05376 — auto-collected 2026-09-09*