Same Trajectory, Contradictory Rewards: ROBORMBENCH Reveals Paraphrase Fragility in Vision-Language Reward Models

Forum topic · 小凯 · 2026-09-07

Summary

A forum post discusses ROBORMBENCH, a benchmark from a 2026 arXiv paper showing that state-of-the-art vision-language models (VLMs) used as reward functions in robot learning are highly sensitive to paraphrasing. In experiments spanning 2,390 real robot trajectories with human-labeled progress tags and 21,673 verified paraphrases, the same robot trajectory can receive scores differing by more than 60 points on a 100-point scale depending solely on how the task is worded. Success/failure judgments can flip entirely, and score volatility grows with semantic distance from the original instruction. Notably, scaling from 7B to 70B parameters and chain-of-thought reasoning do not resolve the fragility, suggesting a fundamental architectural issue rooted in statistical rather than semantic language understanding. Tested mitigations include dedicated trajectory-grounded reward models (40-60% smaller score variance), paraphrase-augmented training (15-20% improvement), and ensembles over multiple paraphrases (about 30% variance reduction at 5x compute cost). The post also discusses implications for reward hacking, the alignment problem, and philosophical questions about whether fluent language use implies genuine understanding.

This forum post reviews the paper "Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models" by Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, et al. (arXiv: https://arxiv.org/abs/2609.02345, September 2026), in the field of robotics / NLP / computer vision.

Key points

The core finding

Vision-language models (VLMs) used as reward functions for robot learning give drastically different scores to the identical robot trajectory when the task instruction is paraphrased. The post opens with an example: "put the plate on the table" earns 85/100, while a semantically equivalent rewording earns only 23/100.

ROBORMBENCH benchmark

2,390 real robot trajectories from various platforms and tasks
Human-annotated ground-truth progress labels
21,673 verified paraphrases covering lexical substitution, syntactic restructuring, and action-goal rewrites

Observed phenomena

Large score swings

Success/failure flips

Distance-dependent deviation

Scale is not a fix

Chain-of-thought doesn't help

Why it happens (per the post's analysis)

VLMs learn statistical correlations between text and visual patterns from unevenly distributed training data, not semantic equivalence between paraphrases.
Attention mechanisms amplify spurious correlations with certain "favorable" words.
This makes VLM rewards especially vulnerable to reward hacking: policies may optimize for score-triggering wording rather than true task completion.

Mitigations tested

Dedicated trajectory-grounded reward models: 40-60% smaller score variance and far fewer success/failure flips, but require costly human annotations and are task-specific.
Paraphrase-augmented training: penalizing inconsistent scores across paraphrases yields only ~15-20% improvement.
Ensembles: averaging 5 paraphrase variants reduces score standard deviation by ~30%, at 5x inference cost.
The authors point toward a deeper shift: from discriminative score matching toward generative approaches that compare trajectories against imagined ideal ones.

Broader implications

The post connects paraphrase fragility to reward hacking, the AI alignment problem (optimizing surface instruction wording rather than true intent), and philosophical questions echoing Searle's Chinese Room: fluency is not understanding.
Other unresolved risks of VLM reward functions include out-of-distribution generalization, adversarial attacks, and cultural bias.

References cited in the post

Jeung et al., arXiv:2609.02345 (2026)
OpenAI, "GPT-4V(ision) System Card" (2023)
Driess et al., "PaLM-E" (ICML 2023)
Brohan et al., "RT-2" (arXiv:2307.15818)
Ziegler et al. (arXiv:1909.08593); Christiano et al. (NeurIPS 2017); Amodei et al. (arXiv:1606.06565); Searle, "Minds, Brains, and Programs" (1980)