Rephrase Before You Act: Mitigating Instruction Phrasing Sensitivity in Vision-Language-Action Models

Forum topic · 小凯 · 2026-10-09

Summary

This arXiv paper (2610.10526) reveals that vision-language-action (VLA) models are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-language models they are built on. A single-word edit can swing success rates by tens of points: pi-0.5 turns on a LIBERO stove 100% of the time for "switch on the stove" but only 2% for "switch on the hot plate", and even a pi-0 checkpoint finetuned with rephrase augmentation shows swings up to 61 points. The authors characterize this sensitivity using statistically tested single-edit swings and oracle phrase search, showing phrasing alone nearly closes the 21-point gap between in-distribution and out-of-distribution tasks. Their mitigation approach keeps the policy frozen: they score multiple phrasings of a few training tasks, use an LLM to distill the evidence into 10-20 rephrasing rules, and rewrite each incoming instruction once at deployment. These rules improve frozen pi-0's relative performance by 16-27% across 12 held-out tasks, with gains concentrated on out-of-distribution tasks, and boost pi-0.5's in-distribution LIBERO success from 93.6% to 97.8%. The method requires no retraining, no per-step verification, and applies zero-shot to unseen tasks and instructions.

Paper Overview

Field: NLP / Robotics
Authors: Mikey Watts, Yuchen Cui
Published: 2026-10-07
arXiv: 2610.10526

Summary

Vision-language-action models (VLAs) are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-language models they are built on. A one-word edit can move success by tens of points: π₀.₅ turns on a LIBERO stove 100% of the time for "switch on the stove" and 2% for "switch on the hot plate", and a π₀ checkpoint finetuned with rephrase augmentation still shows swings of up to 61 points.

The authors characterize this sensitivity with statistically tested single-edit swings and an oracle phrase search, which shows that phrasing alone nearly closes the 21-point gap between in-distribution and out-of-distribution tasks.

They then reduce this sensitivity without modifying the policy. Because the sensitivity is systematic, it can be expressed as explicit rules: they score multiple phrasings of a small number of training tasks, have a large language model distill the evidence into 10-20 rephrasing rules, and at deployment rewrite each incoming instruction once according to these rules.

Key Results

The distilled rules improve frozen π₀'s relative performance by 16% to 27% across 12 held-out tasks (covering adversarial, VLM-generated, and human-generated phrasings), with gains concentrated on out-of-distribution tasks.
The approach also works for π₀.₅ and on LIBERO, raising in-distribution finetuned success from 93.6% to 97.8%.
Requires no retraining and no per-step verification, and applies zero-shot to unseen tasks and instructions.

Original Abstract (excerpt)

> Vision-language-action models (VLAs) are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-language models they are built on. A one-word edit can move success by tens of points: π₀.₅ turns on a LIBERO stove 100% of the time for "switch on the stove" and 2% for "switch on the hot plate", and a π₀ checkpoint finetuned with rephrase augmentation still shows swings of up to 61 points. We characterize this sensitivity with statistically tested single-edit swings and an oracle phrase search, which shows that phrasing alone nearly closes the 21-point gap between in-distribution and out-of-distribution tasks...

Full paper: arXiv:2610.10526

Rephrase Before You Act: Mitigating Instruction Phrasing Sensitivity in Vision-Language-Action Models

Summary

Paper Overview

Summary

Key Results

Original Abstract (excerpt)

Tags