Same Trajectory, Contradictory Rewards: ROBORMBENCH Reveals Paraphrase Fragility in VLM Reward Models

Forum topic · 小凯 · 2026-09-08

Summary

A new arXiv paper (2609.05401) examines vision-language models (VLMs) used as reward functions for robotic learning. Effective reward models should be paraphrase invariant: the same robot trajectory should receive the same progress score regardless of how the goal is phrased. The authors show that current proprietary and open-source VLM reward models frequently violate this property. Simply rephrasing the instruction can substantially change predicted progress scores and even flip identical robot behavior between failure and success. To quantify this failure mode, the team introduces ROBORMBENCH, a benchmark containing 2,390 real-robot trajectories with ground-truth progress labels and 21,673 verified paraphrases covering lexical, syntactic, and action-goal rewrites. Experiments reveal that paraphrase-induced instability is widespread and severe across models, and grows under more diverse rewrite types. The paper, by Wonje Jeung and colleagues, was published on arXiv in September 2026.

Paper Overview

Field: NLP / Robotics Authors: Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, Yoonjun Cho, Dongjae Jeon, Bumjun Kim, Jean Oh, Youngjae Yu, Albert No Published: 2026-09-04 arXiv: 2609.05401

Abstract (translated from the original)

Vision-language models are increasingly used as reward functions for robotic learning, but this role requires paraphrase invariance: the same trajectory should receive the same reward under semantically equivalent goal descriptions. The authors show that current VLM reward models often violate this property. Paraphrasing the instruction alone can substantially change predicted progress scores, and can even flip identical robot behavior between failure and success.

To measure this failure mode, they introduce ROBORMBENCH, a benchmark with:

2,390 real-robot trajectories
Ground-truth progress labels
21,673 verified paraphrases spanning lexical, syntactic, and action-goal rewrites

Across proprietary and open-source VLMs, paraphrase-induced instability is widespread and severe, and grows under more diverse rewrite types. The findings highlight a fundamental robustness gap that must be addressed before VLM-based reward models can be reliably deployed in robot learning pipelines.

Resources

Paper: https://arxiv.org/abs/2609.05401

*Auto-collected on 2026-09-08.*