TAPDreamer: Transferable Adversarial Patches That Break World Action Models Without Target Queries

Forum topic · 小凯 · 2026-10-07

Summary

TAPDreamer is an adversarial attack against world action models that uses only a publicly available encoder to construct a fixed, localized visual patch capable of transferring across tasks and action architectures without querying the target policy. The key insight is that the interaction between patch-induced changes in attention weights and value vectors broadcasts a nearly identical representation shift beyond the patch footprint, and this shift remains stable across task observations. TAPDreamer optimizes the patch by maximizing the global L1 distance between clean and patched encoder representations using six frames from a single source task. In closed-loop evaluation, one frozen patch covering roughly 6.5% of the input drops FastWAM's success rate from 97.7% to 0.0% across 40 LIBERO tasks and from 90.8% to 0.0% across 50 RoboTwin tasks, while matched random patches retain 81.5% and 79.2% success. The same patch also degrades DreamWAM configurations to 2.1% and 0.8%, and Motus to 10.0%. The findings show that defending only downstream action generation is insufficient: world action model defenses must also protect the shared visual encoder against persistent local perturbations. Paper: arXiv 2610.06814.

Paper Overview

Research area: Adversarial ML
Authors: Xuanyu Lu, Fengqing Jiang, Kaiyuan Zheng, Yichen Feng, Yaorui Ding, Yuetai Li, Zhen Xiang, Bhaskar Ramasubramanian, Basel Alomair, Luyao Niu, Radha Poovendran
Published: 2026-10-05
arXiv: 2610.06814

Summary

World models learn to predict how environments evolve, making them an important foundation for general-purpose robot control. However, world action models depend on camera inputs, and manipulating camera inputs can corrupt the visual representations shared across tasks and action policies. Existing attacks against these models are optimized against the victim's actions or predicted futures, and therefore require access to the target model's outputs.

The authors propose TAPDreamer, an attack on world action models that uses only a public encoder to build a fixed local perturbation. This perturbation transfers across tasks and action architectures without any queries to the target policy.

Core Insight

The interaction between patch-induced changes in attention weights and value vectors broadcasts a nearly identical representation shift beyond the patch's footprint, and this shift remains stable across observations from different tasks.

Based on this insight, TAPDreamer uses six frames from a single source task to maximize the global L1 distance between clean and patched encoder representations.

Results (Closed-Loop Evaluation)

Each benchmark uses a single frozen patch covering roughly 6.5% of the input:

| Benchmark | Clean success rate | With TAPDreamer patch | Matched random patch | |---|---|---|---| | FastWAM, 40 LIBERO tasks | 97.7% | 0.0% | 81.5% | | FastWAM, 50 RoboTwin tasks | 90.8% | 0.0% | 79.2% | | DreamWAM (config 1) | — | 2.1% | — | | DreamWAM (config 2) | — | 0.8% | — | | Motus | — | 10.0% | — |

Implications

These results indicate that protecting downstream action generation alone is insufficient: defenses for world action models must also protect the shared visual encoder against persistent local perturbations.

--- *Auto-collected on 2026-10-07*

TAPDreamer: Transferable Adversarial Patches That Break World Action Models Without Target Queries

Summary

Paper Overview

Summary

Core Insight

Results (Closed-Loop Evaluation)

Implications

Tags